Router and methods for distributed virtualization

ABSTRACT

A router for use in a network includes a scalable architecture and performs methods for implementing quality of service on a logical unit behind a network port; and for implementing storage virtualization. The architecture includes a managing processor, a supervising processor; and a plurality of routing processors coupled to a fabric. The managing processor has an in-band link to a routing processor. A routing processor receives a frame from the network, determines by parsing the frame, the protocol and logical unit number, and routes the frame to a queue according to a traffic class associated with the logical unit number in routing information prepared for the processors. An arbitration scheme empties the queue in accordance with a deficit round robin technique. If a routing processor detects the frame&#39;s destination is a virtual entity, and so is part of a virtual transaction, the router conducts a nonvirtual transaction in concert with the virtual transaction. The nonvirtual transaction accomplishes the intent of the virtual transaction but operates on an actual network port, for example, a storage device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional patent application of and claimspriority to U.S. patent application Ser. No. 10/120,266, filed on Oct.18, 2001, now U.S. Pat. No. 7,200,144 by William C. Terrell, et al.

FIELD OF THE INVENTION

Embodiments of the present invention relate to improved networks havingrouters that perform routing functions and to methods for routingnetwork traffic.

BACKGROUND OF THE INVENTION

In a conventional network, data is transferred between computers andperipherals to accomplish the data processing demands of the computersand peripherals. Demands for data to be transferred via the network mayarise in any particular computer or peripheral in a mannerunsynchronized with demands that arise on other computers andperipherals of the network. Data transfer to accomplish delivery isgenerally between respective ports of the computers and peripherals andmay pass through switches having ports as well. Such switches havenumerous ports and generally retransmit data (also called routingnetwork traffic) from one port to another according to addressinformation associated with the data to be transferred. A pair of portscommunicate via a link between the ports.

Demands generally vary widely in the amount of data to be delivered overthe network and the manner in which the delivery is to be made. Forexample, some demands may be made for a relatively large amount of datawithout regard to the order in which the data is delivered via thenetwork. Other demands may require that the data be delivered in aparticular order. Some demands may have no use for data that ispresented outside of an expected time for delivery. Other demands may bemet at any time, though system efficiency may suffer if delivery is madeoutside of an expected time for delivery.

With a large number of network links, use of the network may beregulated to some extent by establishing a priority for each link. Inparticular, when attempts to meet demands result in delivery of data inbursts between pairs of computers and/or peripherals, networkperformance may exhibit several undesirable results. Network capacity(sometimes colloquially referred to as bandwidth) for servicing lowerpriority links may be unavailable. Delivery of data may be noticeablydelayed. More out of order deliveries may be made. And, service betweenports on particular links may be denied intermittently, causing queuesto fill and network capacity to be used for overhead messages regardingthe control of network traffic as opposed to actually routing thetraffic.

Traditional approaches to improving a network's ability to deliver datawhich would otherwise be delivered in bursts and to decreasing thelikelihood of the undesirable results described above have focused onincreasing network data transfer speed, increasing the depth of queuesfor data awaiting processing before or after transfer via the network,and increasing the instruction processing speed for processors (e.g.,per-port processors) that accomplish delivery over the network. In aconventional architecture, each port may be implemented with a processorand memory dedicated to servicing all forms of traffic for that port.

In another known approach to solving some of the problems discussedabove, a traffic stream having a traffic profile is affected byprovisioning a facility for traffic conditioning as described in RequestFor Comment “An Architecture for Differentiated Services,” RFC2475 by S.Blake of Torrent Networking Technologies. A traffic profile is a set ofdesired temporal properties for a traffic stream (i.e., packet rate andburst size). A traffic stream is an administratively significant set ofmicroflows that traverse a path segment as selected by a particularclassifier. Provisioning includes mapping traffic streams to per hopbehaviors, and specifying methods of traffic conditioning. Per hopbehaviors are effected by shaping. Traffic conditioning is defined asclassifying, metering, marking, shaping, and dropping packets. Amicroflow classifier selects packets (e.g., for marking) based on anarbitrary number of header fields including source address, destinationaddress, protocol (e.g., IP), fields (e.g., DS field in IP header),source port, and destination port. Marking is defined (for IP) assetting the value of the DS field. Metering is defined as measuringtemporal properties of a traffic stream. Shaping is delaying packets toconform a traffic stream to a desired traffic profile. Shaping includesenqueueing a marked packet and holding the packet in queue untiltransmitting the packet would not exceed a desired traffic profile. Thebasic architecture assumes that traffic conditioning functions areaccomplished at each ingress and egress node (i.e., at each port of anedge node) of the network. According to a first conventional hardwarearchitecture, all traffic conditioning functions would be accomplishedby a central processing unit (CPU) serving a group of ports at aningress and egress node. Such a CPU would not be capable of significantbandwidth. According to a second conventional hardware architecture,each port of an edge node would be implemented with a processor andmemory dedicated to performing traffic conditioning functions byservicing all forms of traffic for that port.

A large portion of network traffic is associated with reading or writingdata storage media. The data delivery problems described above areevident in networks that provide shared access to data storage devices.Managing data for improved access according to traditional approacheshas included introducing servers between data storage devices and thenetwork. Such server technology impedes network traffic flow, and mayfacilitate unexpected denial of access or damage to data due to failuremechanisms with a single point of failure.

Without the present invention, data delivery cannot be further improvedwithout unreasonably increasing the cost per port of the network and thecomputers and peripherals that use the network. Increased costs stemfrom increased memory for queues and sophisticated processinginstructions to be executed by the port processors, from increasedprocessing speed, and from circuits that operate at higher frequenciesto provide increased network data transfer speed. The comparatively highcost of circuits that operate at increased frequency stems fromdifficulties in designing such circuits and difficulties in fabrication.

SUMMARY OF THE INVENTION

A router, in one embodiment of the present invention, routes frames in anetwork. The router includes means for participating as a virtual targetin a virtual transaction initiated by an initiator of the network andmeans for initiating a nonvirtual transaction with a target of thenetwork to accomplish an intent of the virtual transaction.

By analyzing at least a portion of a received frame, and preparing anoutbound frame back to the requester, a router operating according tovarious aspects of the present invention provides a logical interfacebetween the requester and resources. An additional outbound frame to aresource may be prepared by the router to fulfill the request. A logicalinterface facilitates management of the resources for improvedefficiency and reliability of data transfers; and, supports demandinglevels of quality of service as to order and timeliness of deliveries.

In another embodiment a router includes a processor that stores avirtual resource identifier and routes a frame that includes indicia ofa nonvirtual resource identifier. The nonvirtual resource identifier maybe determined by the processor with reference to an association betweenthe nonvirtual resource identifier and the virtual resource identifier.The association may be made by an administrating process andcommunicated to the processor as routing information.

A router, in another embodiment of the present invention, includes aprocessor that stores a resource identifier determined from a firstframe and routes a second frame in accordance with the resourceidentifier. For example, the second frame may be received withoutindication of the resource identifier and received after the first frameis received.

In another embodiment of the present invention, a router includes aprocessor that routes a frame in accordance with a policy value toimplement a quality of service. The policy value is determined at leastin part by parsing the frame to determine a resource identifier andrecalling an association of the policy value and indicia of the resourceidentifier. The association may be made by an administrating process andcommunicated to the processor as routing information.

By analyzing at least a portion of a received frame, and identifyingmore than one field value, a router operating according to variousaspects of the present invention selectively controls the quality ofservice as applied to particular data transfers and frames havingparticular sets of field values. Quality of service may effectively becontrolled for a predetermined protocol and/or predetermined group ofresources. Quality of service may include specifications regarding orderand timeliness of deliveries, or in other words, bandwidth allocation,maximum delays, and reduction in network congestion. Statistics may becollected and analyzed for a subflow.

A router, in another embodiment of the present invention, includes amanaging processor, a supervising processor, and a routing processor.The managing processor performs a proxy process that responds to acontrol frame directed to a virtual entity. The supervising processorperforms a control process that responds to a control frame directed tothe router. The routing processor routes data frames directedrespectively to virtual and to nonvirtual entities via the network.

A router, in another embodiment of the present invention, includes twoprocessors. The first processor performs a proxy process for a virtualmember of the network. The proxy process responds to a control framehaving a first network port identifier. The virtual member correspondsto at least one nonvirtual member or resource of the network. Thenonvirtual member responds to a data frame having a second (i.e.,different) network port identifier. The second processor performs arouting process that routes frames having the first network portidentifier to the proxy process, routes frames having the second networkport identifier to the nonvirtual member, and on receiving a data framehaving the first network port identifier, routes a substitute data framehaving the second network port identifier. For example, data framesoriginally addressed to the virtual member are readdressed and routed toa corresponding nonvirtual (e.g., actual) member.

The modular architecture provided according to various aspects of thepresent invention permits scaling of the router design and scaling ofthe network, lowering the cost for competitive router products andimproving network maintenance.

A router, in another embodiment of the present invention, routes a framereceived from a network. The router includes a routing processor. Therouting processor includes: a frame processor, a parser, a plurality ofqueues, a submitter, and a memory circuit. The parser prepares a flowlookup in response to the frame received from the network. The memorycircuit performs a flow lookup and provides a result as directed by thesubmitter and a first entry in a first queue, the first entry havingbeen enqueued by the parser. The memory circuit also performs a subflowlookup and provides a result as directed by the submitter and a secondentry, the second entry having been enqueued in a second queue by theframe processor in accordance with the result of the flow lookup. Theframe processor routes the frame in accordance with the result of thesubflow lookup.

A router, in another embodiment of the present invention, includes: aplurality of physical ports, a managing processor, and at least onerouting circuit coupled to the manager by a first bus. Each routingcircuit includes: a supervising processor, a memory, a second bus, and aplurality of port logic circuits. The memory includes indicia of arouting table. The memory is coupled to the supervising processor by thesecond bus. The plurality of port logic circuits is coupled to thesupervising processor by a third bus. Each port logic circuit provides amultiplicity of the physical ports. Each port logic circuit is coupledto other port logic circuits for data transfer between physical ports.At least one physical port of the plurality is coupled to the managingprocessor.

By providing in-band access to the managing processor, virtualizationfunctions are less complex and more efficient. Wire speed virtualizationis facilitated.

A router, in another embodiment of the present invention, includes: aplurality of physical ports; a managing processor having a first memory;and at least one routing circuit coupled to the managing processor by afirst bus. Each routing circuit includes a supervising processor and aplurality of port logic circuits. The supervising processor has a secondmemory. Each routing circuit further includes a third memory. The thirdmemory includes indicia of a routing table. The third memory is coupledto the supervising processor by a second bus. The plurality of portlogic circuits are coupled to the supervising processor by a third bus.Each port logic circuit provides a multiplicity of the physical ports.Each port logic circuit is coupled to other port logic circuits for datatransfer between physical ports. Each port logic circuit has a frameprocessor that includes a respective fourth memory. The managingprocessor updates the second memory via the first bus. The supervisingprocessor updates the third memory and the fourth memory via the secondbus.

By loading and updating routing information tailored to particular frameprocessors and tailored to particular routing processors, thecomputational burden of performing virtualization functions may bedistributed among routers of a network.

A method, in another embodiment of the present invention, is performedby a router for routing frames in a network. The router includes aplurality of network ports, a fabric, and a plurality of routingprocessors coupled between the fabric and the network ports. Eachrouting processor includes an ingress buffer, for receiving frames froma network port and for transmitting frames to the fabric; an egressbuffer for receiving frames from the fabric and for transmitting framesto the network port; and a frame processor. On receiving from arequester a data frame directed to a virtual participant, the frameprocessor modifies the data frame in the ingress buffer for routing to anonvirtual participant. On receiving from the fabric a data frame notdirected to a nonvirtual requester, the frame processor modifies thedata frame in the egress buffer for routing to a nonvirtual requester.Further, the frame processor may, on receiving from the fabric a dataframe not directed to a nonvirtual requester for which the frameprocessor does not have sufficient modification information, route thedata frame via the fabric to another routing processor of the plurality.

A router, in another embodiment of the present invention, includes afirst routing processor and a second routing processor and a fabric.Each routing processor includes: an ingress buffer coupled to an inputport, an egress buffer coupled to an output port, a parser, and a memorythat stores routing information. The ingress buffer is coupled betweenthe input port and the fabric to transfer frames from the ingress bufferto the fabric. The egress buffer is coupled between the fabric and theoutput port to transfer frames from the fabric to the output port. Thefirst routing processor parses a frame received from its input port todetermine a virtual destination identifier, determines a nonvirtualtransaction identifier in response to the virtual destinationidentifier, prepares a second frame having the nonvirtual transactionidentifier, and transmits the second frame to the fabric. The secondrouting processor receives the transmitted second frame from the fabricand transmits the second frame to its output port. The second processor,on receiving a third frame on its input port parses the third frame todetermine a nonvirtual transaction identifier, marks the third frame formodification, and transmits the third frame to the fabric. The firstprocessor receives the transmitted third frame from the fabric, parsesthe third frame to access the routing information from its memory,modifies the third frame in accordance with the accessed routinginformation, and transmits the modified frame from its output port.

By operating on frames in the ingress buffer and egress buffer, a lowercomplexity router design results. For example, less memory is needed formaintaining virtual context tables.

A router, in another embodiment of the present invention, includes aplurality of ports and a routing processor. The routing processorincludes: at least a portion of a fabric, an ingress buffer, an egressbuffer, The ingress buffer is coupled between the fabric and a firstport of the plurality to transfer frames from the fabric to the firstport. The egress buffer includes a plurality of queues, an arbitratingcircuit coupled between the egress buffer and the first port, and acounter associated with each queue. Each counter has a respectivecurrent count. The arbitrating circuit (a) adds received grants to agrant pool for the plurality of queues; (b) transfers a frame from aselected queue to the fabric when sufficient grants exist in the grantpool; (c) decrements the grant pool in response to the transfer; (d)adds transmitted frame size to the counter associated with the selectedqueue; and tests whether the counter associated with the selected queueis greater than a threshold. If so, the arbitrating circuit: (a) sets anoverrun amount to the current count of the counter associated with theselected queue; (b) resets the counter associated with the selectedqueue; (c) subtracts the overrun amount from a current count of eachother counter; (d) clears all asserted stalled flags; and (e) stalls theselected queue.

A router, in another embodiment of the present invention, includes aplurality of routing processors each having at least a portion of adistributing circuit. Each distributing circuit portion has a crossbarswitch that completes a plurality of point-to-point connections betweenrouting processors. The crossbar switch operates in response to at leastone of: an input that indicates a number of routing processors that havebeen installed, and an input that indicates a position of the routingprocessor among the number of routing processors. A second crossbarswitch may provide a termination for a point-to-point connectionaccording to at least one of: an input that indicates a number ofrouting processors that have been installed, and an input that indicatesa position of the routing processor among the number of routingprocessors.

Combinations of the various aspects of the present invention providesolutions to the problems described in the background section andmitigate other problems. For example, stall and continue capabilities ona subflow basis accommodate bursty network traffic from variousapplications sharing a network link. Further, accommodating quality ofservice differences (e.g., in the time or ordering of data) on a subflowbasis better accommodates performance variations among processes andstorage functions in any member or within the network (e.g., aninterswitch link). A router operating according to various aspects ofthe present invention efficiently allocates bandwidth without completelystalling a low priority flow or unreasonably fragmenting a high priorityflow. Routers that provide a logical resource interface provide moreefficient and more reliable networks for application service providersand storage service providers, thereby lowering the cost of operatingand lowering the cost of these services to the consumer.

According to various aspects of the present invention, sophisticatednetwork functions are accomplished without a general purpose processorper port. Such functions include, inter alia, mirroring, third partycopy, arbitration based on subflows, subflow stalls, statisticsgathering, provision of a logical resource interface, and maintainingcaches in the router for read and write operations.

By maintaining one or more pointers to the original copy of a snapshotand possibly to revised portions of the snapshot, the time to initiallysupport use of a snapshot may be reduced and the interruption due totaking time to prepare a full copy of the snapshot may be avoided.

By maintaining a cache in the router, more efficient data transfer to amember of the network results. Egress from the cache is provided to meetthe needs of the resource as opposed to the resource being forced toaccommodate operation of the network or operation of another networkmember.

By maintaining a cache in the router, a multicast write is accomplishedwith fewer data transfers. More efficient network operation results.

In a router architecture according to various aspects of the presentinvention, memory is provided where it can be effectively used and thecost of router circuits can be decreased by avoiding large amounts ofmemory that are infrequently accessed. Operations limit the need tosynchronize redundant copies of information in separate parallelprocessors within the router. Such an architecture supports framedisposition at the maximum rate on all ports and full mesh connectivityat wire speed. Routers based on scaling and reusable design (e.g., areconfigurable full mesh circuit) help control the overall router costand reduce dependency on higher cost processors and memory. Furthermore,routers with different quantities of ports may be economically assembledwith a greater reliance on common designs and subassemblies, loweringthe cost of manufacturing.

According to various aspects of the present invention, processors thatare in the data path execute frame preparation functions with referenceto commands and information prepared by processors that are not in thedata path. Such functions include, for example, access control from acentralized administration processor; providing security from rogueprocesses (e.g., identifier translation tables (e.g., used for resourcemapping or frame routing) are not directly accessible from the portinterface); or gathering statistics on a subflow so that controldecisions may be based on use of the network by a particular type ofprocess (e.g., Virtual Interface (VI) communication having priority overSCSI communication from the same port of the network) or a particulartype of storage device (e.g., streaming audio access having priorityover data processing transactional file access). A managing processor ina router may filter statistics and more efficiently report to anadministrating processor for management of virtual resources.

An administrating processor updates the configuration (e.g., routingtables) of several routers uniformly. An administrating process mayassign network port identifiers to be used for virtual members, virtualresources, and proxy processes. Proxy processes may receive controlframes for a virtual member or virtual resource.

As router products are developed with varying need for processing, theratio of the various processors to the number of ports may beeconomically scaled while continuing to benefit from the investment incircuit and firmware design. The following are but a few examples. Thenumber of buses made active in the mesh may scale with the quantity andbandwidth of the ports. Port protocol support may be downloaded to theprocessor(s) responsible for particular ports. Supervisory processingmay scale with the quantity of ports in part due to the bus interfacebetween a plurality of port processing slices and the supervisoryprocessor(s). Managing processor(s) scale with the number of ports inpart due to use of one or more in-band links to the supervisingprocessor(s). Processing responsibility scales with the amount ofavailable memory due in part to the shared nature of memory between portlogic circuits. RAID device control may be implemented at the devicecluster level from processor(s) in one or more routers or fromprocessor(s) that is(are) part of a member. Multiple protocol capabilityscales with different demands for different protocols. Multiple zonecapability for load balancing scales by performance and extent ofphysical, logical, and virtual resources.

A router according to various aspects of the present invention detectsin a virtual data frame a page boundary crossing, initiates nonvirtualdata frames to accomplish the operation intended, and routes thenonvirtual data frames to corresponding nonvirtual storage. A pageboundary crossing occurs, for example, when reference is made in a dataframe to a portion of a virtual storage device, and the reference whenmapped to nonvirtual storage would include more than one page of one ormore nonvirtual storage devices.

By detecting page boundary crossings and initiating data frames, arequester may operate on a virtual resource without knowledge of thestructure and organization of the corresponding nonvirtual resource,simplifying such operations from the point of view of the requester.

BRIEF DESCRIPTION OF THE DRAWING

Embodiments of the present invention will now be further described withreference to the drawing, wherein like designations denote likeelements, and:

FIG. 1 is a functional block diagram of a system according to variousaspects of the present invention;

FIG. 2 is a data flow diagram of processes in the system of FIG. 1;

FIG. 3 is a data flow diagram of the administrating process of FIG. 2;

FIG. 4 is a data flow diagram of the managing process of FIG. 2;

FIG. 5 is a data flow diagram of the supervising process of FIG. 2;

FIG. 6 is a data flow diagram of the routing process of FIG. 2;

FIGS. 7, 8, 9, and 10 form a flow chart of a method for routingaccording to various aspects of the present invention;

FIG. 11 is a functional block diagram of a router of the system of FIG.1;

FIG. 12 is a functional block diagram of the supervising processor ofFIG. 11;

FIG. 13 is a functional block diagram of the memory circuit of FIG. 11;

FIG. 14 is a functional block diagram of the port logic circuit of FIG.11;

FIG. 15 is a functional block diagram of the descriptors of FIG. 14;

FIG. 16 is a message sequence diagram for operations performed by thesystem of FIG. 1;

FIGS. 17-20 form a flow chart of methods performed by router 102 of FIG.2;

FIG. 21 is a functional block diagram of a fabric having five fabricnodes according to various aspects of the present invention;

FIG. 22 is a functional block diagram of the fabric of FIG. 20implemented with three fabric nodes;

FIG. 23 is a functional block diagram of the distributing circuit ofFIG. 11; and

FIG. 24 is a message sequence diagram for operations performed by thesystem of FIG. 1.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

A system according to various aspects of the present invention mayinclude any computing environment supporting transfer of data amongcomputer systems via a communication network. Such a system, in oneimplementation, provides more efficient non-blocking delivery of data,improved utilization of bandwidth, a facility for managing networktraffic flows, subflows, and virtual flows, and higher quality ofservice. Data may be transferred between application programs beingexecuted by one or more of the computer systems, between an applicationprogram and a data storage device, or between one or more data storagedevices.

The network may be understood as a graph or a tree having network nodes.A communication network of the present invention includes at least onecomputer system at each of several network nodes. Each network node iscoupled by a link from time to time for communication with other networknodes. Each link includes conventional computer communication technologyat the physical layer and primitive layers of the type including, forexample, local area, wide area, dedicated telephone, wireless, andsatellite services and including conventional data communicationhardware and software at each network node. The popular computernetworks known as storage area networks, intranets, the Internet, theWorld Wide Web, and the National Information Infrastructure are examplesof communication networks in which various aspects of the presentinvention may be practiced. Network nodes are generally at physicallyseparate locations and are generally suitably identified, for example,by a node name, node identifier, node address, a world wide identifier(WWPN), a uniform resource locator (URL), a name from a domain namesystem (DNS), or an Internet Protocol address (IP).

Data transfer at the lowest level occurs via a link between ports,nominally a requesting port and a participating port, where a requestingport requests a data transfer and the participating port either suppliesthe data (e.g., a read) or receives the data being transferred (e.g., awrite). A port includes a physical implementation for common signalingbetween ports (e.g., any circuitry suitable for the transfer media); anda logical implementation (e.g., any combination of firmware andsoftware). Cooperation between ports occurs in accordance with aphysical protocol (e.g., signals and their characteristics) and alogical protocol (e.g., one or more layers of application programinterfaces). The physical and logical implementations and protocolstogether constitute a port by which other software can manage, amongother things, how to obtain the data to be supplied to the port and whatto do with the data obtained from the port. A port may communicate usingseveral protocols. Frames according to a first protocol may beencapsulated (i.e., become the payload) in frames according to anotherprotocol. Ports of a router according to various aspects of the presentinvention may support, for example, combinations of Fibre Channel (FC)Protocol (FCP), Internet Protocol (IP), based IEEE 802.3 Ethernetprotocol, Small Computer Systems Interface (SCSI) Parallel Interface,Serial Bus Protocol, IEEE 1384 (Fire wire), SSA SCSI-3 Protocol,Scheduled Transfer, and Virtual Interface (VI).

A network node may include one or more ports. Multiple ports at anetwork node may be serviced as a group (e.g., a hunt group) to serve anupper level process with higher band width, to provide fail-overcapability, or to serve multiple parallel processes. Network nodeidentifiers (e.g., port identifiers) facilitate requesting (e.g.,initiating) and participating in a data transfer (e.g., performing as atarget or as a virtual target).

A group of ports may provide data transfer functions transparently. Forexample, a bridge, located between a requester and a participant, mayreceive requests in a first protocol (e.g., not understood by theparticipant) and provide a corresponding request to the participant in asecond protocol (e.g., not understood by the requester). Further, arouter, located anywhere in the network, may serve as a hub for severallinks; each link being served by one or more ports. Such a router routesnetwork traffic between a requester (having a requesting port) and aparticipant (having a participating port) without either the requesteror the participant having knowledge of the port identifiers of the portsof the router. The router transfers traffic between its ports inaccordance with a routing table that defines communication paths throughthe router. The routing table may be specified by a network technician,by an administrator as discussed below, or may be determined by therouter as a result of communication with other routers to which it islinked. A router according to various aspects of the present inventionmay function as a gateway receiving frames at an input port according toa first protocol and forwarding frames to an output port that (a)encapsulate the input payload; (b) strip the encapsulation of an inputframe and forward the payload to the output port; or (c) use frames ofthe second protocol to conduct the function intended by the firstprotocol (e.g., data transfer with a virtual destination or with alogical destination; or a Virtual Interface transaction to a SCSItransaction).

A system according to various aspects of the present invention includesa communication network and numerous computer systems. Any of thecomputer systems that are currently members of the communicationnetwork, may transfer data to any other computer systems that aremembers of the communication network (or will be at a suitable futuretime) via links through routers. For example, system 100 of FIGS. 1-6,11-14, and 23 includes communication network 101 (i.e., network 101) andmembers 110-117. Network 101 includes a link to each member:respectively links 150-158 to members 110-117. Network 101 also includesrouters 102-105. The quantity, configuration, and arrangement ofmembers, links, and routers in system 100 is merely illustrative and anynumber, configuration, and arrangement may be used in practice of thevarious aspects of the present invention.

Practice of variations of the present invention is independent ofwhether any particular link is maintained continuously, as in adedicated line, or is maintained for a suitable duration. Members,links, and routers may each incorporate multiple units and be organizedto provide redundancy or fail-over capacity to avoid a single failurefrom disrupting communication.

System administration includes establishing and maintaining routerconfiguration for some or all routers of a network as the utilization ofthe network changes, as link reliability changes, and as the networkgrows or shrinks in number of links, routers, and members. Informationfor manual or dynamic network administration may be collected andreported by routers of the network. Administration may be accomplishedby use of one or more workstations (e.g., for a human operator) orservers (e.g., for administration directed by process(es) running on theservers). Network 101 includes administration subsystem 109 having port106. Link 107 supports communication between administration subsystem109 and any router of network 101, particularly connecting port 106 toport 108 of router 102. Ports 106 and 108 and link 107 may be identicalin structure and function to links and ports described above withreference to routers and members. In an alternate implementation,administration may be accomplished by any suitable member.

System administration may include management of network topology and mayinclude management of virtualization. Virtualization includes thedesignation (e.g., mapping) of a nonvirtual member or nonvirtualresource (e.g., a nonvirtual entity) to be used in place of anyreference to a virtual member or to a virtual resource (e.g., a virtualentity), the communication of that designation to suitable routers, andthe use of that designation in routing packets. Routers according tovarious aspects of the present invention may perform the communicationand use of designations that are defined by system administration.

A member of a network is a computer system that communicates via a linkas described above and either operates, inter alia, to request datatransfer or to participate in data transfer via the link. Some membersmay provide a resource to network 101 so that all members of the networkmay share the capability of the resource. For example, members 110-119may include all or any part of the structure and functions describedbelow with reference to members 115-116. Members 115-116 are capable ofrequesting data from any other member 110-117 or participating in datatransfer with any other member 110-117 and vice versa.

Any member may include a subnetwork. A subnetwork includes any subsystemthat employs ports connected to network 101 for communication generallybetween any member of network 101 and any subnetwork member (e.g., aresource) that is not directly connected to network 101. The interfacebetween network 101 and such a subsystem may provide redundancy,fail-over, multiple or expanded use of network ports, access controls,security (e.g., functions of a conventional firewall), protocolconversion (e.g., functions of a bridge), and/or priority flow controls(e.g., functions of a router as discussed herein). For example, member115 includes subnetwork 170 having ports 165 and 166 connected tonetwork 101 via links 155 and 156 respectively, a port interface 171, aresource interface 172, a controller 173 servicing interfaces 171 and172, a plurality of resources 174 that includes a processing resource175, and a storage resource 177. Port interface 171, resource interface172, and controller 173 may cooperate as a server 178. The plurality ofresources 174 may include zero or more processing devices (e.g.,computers, servers, or workstations) and zero or more storage devices(e.g., disks, tapes, media handlers, or RAID systems).

Port interface 171 performs suitable port interface functions (e.g.,signaling protocols) as described herein and is exemplary of ports160-168 respectively of members 110-117. Port interface 171 isconfigured, directed, and controlled by controller 173. Any conventionalstatus and command interface signaling may couple port interface 171 andcontroller 173. Resource interface 172 performs suitable interfacefunctions (e.g., signaling protocols) to accomplish any conventionalnetwork functions for and among resources via subnetwork 170. Inalternate implementations, interfaces 171 and 172 may be integrated asone interface, may operate in the absence of a controller 173, and/ormay be integrated with one or more resources. Port interface 171 andresource interface 172 communicate over line 176 (e.g., a bus or alink).

Controller 173 accomplishes all conventional protocol functions notalready implemented in port interface 171 and resource interface 172.Controller 173 may include memory used, for example, for programmableoperations of controller 173, data buffering, stateful control ofinterfaces 171 and 172, and subnetwork communication. Controller 173communicates with the plurality of resources 174 via subnetwork 170.Subnetwork 170 may include any conventional logical and physicalorganization. As shown, each resource 175 and 177 communicates withresource interface 172 via a dedicated link. Communication betweenresources and from resources to ports 165 and 166 is accomplished by thecooperation of interfaces 171, 172, and controller 173. Controller 173may perform processing of the type known as Random Array of IndependentDisks (RAID) for one or more storage devices 177. Controller 173 mayperform routing and priority functions for fail-over and load sharingamong processing devices (e.g., functioning as an application serviceprovider) and/or analogous functions among storage devices (e.g.,functioning as a storage service provider).

A resource provides any capability used with data communication. Forexample, processing device 175 and storage device 177 may includeconventional computers, array processors, peripherals, personalcomputers, workstations, telecommunications equipment, disk drives, diskdrive arrays, tape drives, tape drive arrays, printers, scanners, videodisplays and cameras, audio equipment, and measurement instrumentation.Generally, devices 175 and 177 and to some extent controller 173 providefunctions described above as a resource to network 101.

Subnetwork 170 may be any conventional network (e.g., a LAN, SCSInetwork, Fibre Channel network, Integrated Drive Electronics (IDE)network, or a star interface to just a bunch of disks (JBOD)).Communication to and from resources 174 may refer to any suitable deviceidentifiers (e.g., World-Wide Identifiers (WWPNs), logical unit numbers(LUNs), or device addresses).

A router includes any mechanism that provides a logical communicationfacility between a requester and one or more participants. The facilitymay be dedicated (e.g., independent of all other communication throughthe router) or shared (e.g., time multiplexed). When communication isaccomplished by separating data into frames (also called packets),frames may be passed through the facility in order, out of order, withor without regard to a time period specified for transfer, repeated, ordropped. If the facility is of the type conventionally known asnon-blocking, no frame that properly enters the router will be dropped.A facility that is non-blocking at full capacity will drop no frameswhile all its ports operate indefinitely at maximum continuouscommunication link capacity. A router according to various aspects ofthe present invention provides a virtual communication facility alone orin cooperation with other routers.

A frame that enters a router at a given port may exit the router at anyone or more ports including the port from which it entered, as directedby router configuration (e.g., static paths and/or dynamic routingtables). For example, router 102 provides non-blocking communicationamong ports 108, 130-133 respectively supporting links 107, 150 and121-123; router 103 provides non-blocking communication among ports134-137 respectively supporting links 121, 151, 152, and 124; router 104provides non-blocking communication among ports 138-143 respectivelysupporting links 122, 124, 153-155, and 125; and router 105 providesnon-blocking communication among ports 144-148 respectively supportinglinks 123, 125, and 156-158.

A router may serve as a core router or as an edge router. Routers102-103 are illustrated as edge routers because they serve links tomembers 110-112 that are outside of boundary 102. Boundary 102 may bedesignated for security purposes or represent a physical or politicaldivide. Routers 104-105 are illustrated as core routers because allports serve links to members within boundary 102 or serve other routersof network 101. In an alternate network, the ports of a core routerserve no members, only other routers. Analysis of frames for purposes ofdetermining a classification and consequently designating theeffectivity of a suitable policy value may occur at an edge router asopposed to a core router. Effectivity may be implemented, for example,by marking a frame, setting a preference bit, specifying a priorityvalue, setting a preemption bit, or identifying a suitable output queuethat is serviced in a manner that is consistent with a desired qualityof service. Core routers may be programmed to pass traffic without suchanalysis. A router may act as an edge router as to some ports (e.g.,ports 140 and 141 of router 104) and as a core router as to other ports(e.g., 138-139 and 142-143 of router 104).

Interswitch links 121-125 may employ any conventional protocol,including a protocol different from the protocol used between a routerand a member. For example, frames leaving a router's port onto aninterswitch link may include additional information that may be removedbefore the frame is passed on a non-interswitch link. Further, thequality of service (QoS) provided by a router's port to an interswitchlink may be better than the quality of service provided by a router'sport to a non-interswitch link.

Preferably, all of a router's ports have identical port physicalimplementations, for example, for convenience of installation andmaintenance of network 101 as other routers and links are added tonetwork 101. A frame routed on an interswitch link may be marked (at aningress edge router) as discussed above for effecting a policy and suchmarking may be removed (at an egress edge router).

Communication among members and resources, according to various aspectsof the present invention, is supported by a system architecture thatfacilitates expansion and reliability. Expansion includes, inter alia,adding physical assemblies to the system to support additional ports,links, and/or processing capacity as well as to support redundant andfail-over capabilities. Reliability is further enhanced by, inter alia,dividing processing responsibility to avoid processing and communicationbottlenecks, and by modular and reusable procedural, data, and hardwarestructures.

A system architecture is a plan by which system functions are made theresponsibility of particular processes for efficient performance ofsystem functions and for efficient communication among processes. Thesystem architecture is systematically applied as implementations of thesystem are developed and expanded. For example, system architecture 200of FIGS. 2-6 includes administration subsystem 109, network 101, androuter 102 comprising managing, supervising, and routing processes.Implementations of router 102 provide one or more processors forexecuting these processes. Systems employing architecture 200 solve theproblems discussed above (e.g., provide qualities of service), expandand contract without disruption of services, and exhibit extraordinaryreliability.

An administration subsystem includes any computer having a port forcommunication via a link to network 101 and a processor that performs anadministrating process. For example, administration subsystem 106 ofFIGS. 1 and 2 may include one or more servers and/or workstations thatprovide a user interface and a port 106, coupled by link 107 to port 108of router 102.

An administrating process provides routing information to any router ofa network and receives reports from any router of the network. Therouting information provided to a router from an administrating processmay define alternative paths through the network that a router maychoose on a frame by frame basis. When a frame identifies a destinationto which it is to be routed, each router of the network, as aconsequence of receiving routing information from an administratingprocess or from another router, may have one or more alternative pathsthat it may use to route the frame successfully. The router is generallyfree to make the choice of a particular path in accordance with routinginformation and other information, including current traffic conditions.Administrating includes assisting a human operator to develop suitablerouting information for any number of routers of the network. To thatend, the administrating process may also obtain or be automaticallyprovided with information describing current conditions of the network.For example, administrating process 202 receives reports from router 102via network links 107 and provides routing information (e.g., paths) vianetwork link 107 to router 102. By coupling the administrating processto the router via network links, any suitable number and locations ofadministrating processes and administration subsystems may be used toaccomplish reliable access to any or all routers of network 101.Consequently, all administrating functions are scalable to thecomplexity of network 101 as network 101 may expand (e.g., as thequantity of routers to be administered by a particular administrationsubsystem may increase).

In one implementation system administration includes network managementand virtualization management functions that may be performedindependently by different operators. In addition, routers of thenetwork may include conventional routers (e.g., that do not recognizevirtual members and virtual resources) and routers according to variousaspects of the present invention (e.g., that recognize a packet that isdestined for a virtual member or a virtual resource). Virtualizationmanagement includes communicating the designation of a nonvirtual memberor resource to each router that is responsible for implementing anonvirtual transaction corresponding to (e.g., in place of) a virtualtransaction. The router receiving such communication is responsible forrouting packets of the virtual transaction and of the nonvirtualtransaction in accordance with routing information as discussed above.

A router, according to various aspects of the present invention includesscalable processes and scalable interfaces for communication betweenprocesses. Consequently, routers of any suitable complexity (e.g.,number and speed of ports, number of protocols supported, and extent offrame analysis) may be implemented in accordance with architecture 200.For example, router 102 includes managing process 204, supervisingprocess 206, and routing process 208. In operation, managing process 204receives routing information via network 101 and provides routinginformation to supervising process 206 via bus 210; supervising process206 stores routing information in memory 211 from which routing process208 retrieves it; and, routing process 208, routes frames through links214 and 216 to network 101 with reference to routing informationrecalled from memory 211. Frames received from network 101 are generallyhandled by routing process 208 in one of three ways: routing at leastthe payload of the same or corresponding frames to network links viafabric 213 and ports 216, routing at least the payload of the same orcorresponding frames to managing process 204 via ports 201 and 214, andpassing at least the payload of the same or corresponding frames tosupervising process 206 via bus 212.

Particular advantages are realized in a system according to variousaspects of system architecture 200. For example, by providing buses 210and 212 as physical entities, processes 204, 206, and 208 may be hostedby independent processors (e.g., processors having access privilegesover particular resources or separately packaged microprocessors).Consequently, each process 204, 206, and 208 may be hosted (e.g.,provided with suitable resources) in scale with the complexity offunctions performed by router 102. Significant economies resultincluding economies related to modular circuit, firmware, and softwaredesign techniques. For example, one or more managing processes 204 maycommunicate on bus 210 with any number of supervising processes 206. Oneor more supervising processes 206 may communicate on bus 212 and byvirtue of shared access to memory 211 with any number of routingprocesses 208. One or more routing processes 208 may communicate withnetwork 101 via any number of ports 216 (e.g., conveying frames to anyadministrating subsystem and any network member) and communicate withany number of managing processes 204 via ports 201-214. In an alternateimplementation, buses 210 and 212 may be a common entity. In yet anotheralternate implementation, processes 204, 206, and 208 may be performedby fewer than three processors (e.g., one processor or one arrayprocessor) and bus communication may be replaced with conventionalinterprocess communication (e.g., software interrupts, semaphores,common buffers, and multithreading).

An administrating process includes any process that provides a userinterface to a human operator for the purpose of determining routinginformation (e.g., for virtualization management, or network management)to be used in routers of a network and that provides informationregarding network utilization. For example, administrating process 202of FIG. 3 includes edit paths process 302, obtain reports process 304,manage link loads process 310, display link utilization process 312, andport I/O (i.e., input/output) process 306. Routing information may bepresented, stored, and communicated in any suitable form.

Edit paths process 302 creates and revises routing information,automatically and in response to input by a human system operator.Routing information, according to various aspects of the presentinvention, may include any combination of descriptions including: a setof alternate paths through the network, an association of a virtualmember and at least one of a nonvirtual member and a nonvirtualresource, and an association of a virtual resource and at least onenonvirtual resource. Each path and association may be defined to includeseveral links and policy values. Each link may be identified as alogical entity or as a physical entity.

A logical entity (e.g., a logical link, a logical resource, or a logicalmember) may correspond from time to time with one or more physicalentities. By referring to a logical entity, the correspondence betweenthe logical entity and any particular physical entity (or entities) maybe determined dynamically, or in accordance with information that is notavailable at the time that the reference to the logical entity is made.For example, a reference to a logical entity need not be revised inlight of the addition or removal of redundant physical entities.Consequently, system 100 may expand or portions of system 100 may failand the reference to the logical entity remains valid (e.g., does notrequire amendment to continue particular network functions). At thephysical level, a router that has received a frame on one of its portseither routes or drops the frame. Routing includes determining (notnecessarily unambiguously) at least one physical output port to whichthe frame may be directed. If no such output port can be determined (orthe only such output port is not available), the frame is said to bedropped. Dropping a frame (e.g., for lack of information sufficient toroute the frame) accomplishes a denial of access to the member orresource intended to receive the frame.

Routing of frames between members (and resources) is somewhat analogousto sending a letter through the postal system. The letter originallybears the address that the sender believes is the current address of theintended recipient. For example, the sender may live in Ohio and mayaddress a letter to a corporate headquarters in Georgia requesting acopy of the latest specialty catalog. The sender need not have anyknowledge of the street addresses of the post offices or the names oftheir internal departments that may be involved. Suppose that thecorporation has moved its headquarters to Florida and has filed with theGeorgia post office a notice of change of address. When the letter isrouted from the point of deposit into the postal system in Ohio to thepost office in Georgia, the postal workers in Georgia may place theoriginal letter in a surrounding envelope and address the outer envelopeto Florida. The corporation may recognize from the outer envelope orotherwise that the letter is requesting a specialty catalog. Thecorporation may then enclose the outer envelope and its entire contentsin a further enclosing envelope and apply the address of a particularcatalog fulfillment center in Indiana; then redeposit it in the postalsystem. At the fulfillment center in Indiana, all envelopes may bediscarded and the catalog shipped to the requester's Ohio address givenin the letter.

When a router operating according to various aspects of the presentinvention determines that the frame that entered the router must have anadditional address, a frame that encloses (and thereby includes) thereceived frame may be prepared and routed. This is analogous toenclosing the letter in an outer envelope as discussed above. Forexample an in-bound edge router may enclose the frame and an out-boundedge router may discard the outer frame and pass merely the inner frame.Alternately, a router operating according to various aspects of thepresent invention may prepare a frame that contains the payload of theoriginal frame and a different address than originally received. This isanalogous to covering an address on a letter with a sticker that bears aforwarding address. Generally, a frame bears at least one destinationidentifier; an address being one form of an identifier as discussedabove. The several identifiers that may be encountered in operation ofsystem 100 are outlined briefly in Table 1 tracing the routing of arequest for data to be supplied by a resource (e.g., a request from SORTprocess 181 of member 116 for file CITIES from member 115.

TABLE 1 Source or Network Destination Entity Context ServicesIdentifiers Role Member User's Upper level User's handle, Requester(e.g., 116) process (e.g., protocol API record number SORT 181) MemberOperating Operating Operating system (e.g., system API system handle,182) filename Member Logical unit Device Path, logical unit abstractiondriver number, block manager API address range Member Logical unitDevice Logical unit (e.g., as driver API number, page supported bynumber, sector device driver number 183) Member Logical port Port APILogical port Requester, (e.g., 167) identifier Source Member Physicalport Signals Physical port identifier Router Physical port SignalsPhysical port Ingress identifier Router Logical port Router inputLogical port (e.g., 147) port logic identifier Router Virtual unitRouter Virtual unit virtualization number, page API number, sectornumber Router Logical unit Router Path, logical unit abstractionmanagement number, block routing logic address range Router Logical unitRouter Logical unit logic number, page number, sector number RouterLogical port Router Logical port (e.g., 146) output identifier portlogic Router Physical port Signals Physical port Egress identifierMember Physical port Signals Physical port (e.g., 115) identifier MemberLogical port Port API Logical port Participant, (e.g., 166) identifierDestination Member Logical unit Device Logical unit driver API number,page number, sector number Member Logical unit Device Path, logical unitabstraction driver number, block manager address range API (e.g.,controller 173) Member Operating Operating Operating system system APIsystem handle, filename (e.g., CITIES file on disk 177) Member Dataentity Upper level User's handle, protocol API record number

Routing information may be stored in any conventional database such aspaths database 303. In one implementation, paths database 303 includesfor each router a data structure (e.g., one or more files) havingrecords, each record comprising a data structure having fields for adestination identifier as specified in a received frame and one or moreof a list of alternate logical or physical ports of the router to whichthe frame may be routed. Table 2 lists several records of paths database303 describing sets of paths for routers 102, 104, and 105. Thereference numbers in Table 2 identify ports shown in FIG. 1. In recordscorresponding to rows of Table 2, reference numbers would be replacedwith logical port identifiers.

TABLE 2 Router Output port Destination at which of the port as therouter at indicated in frame which the the frame was frame was receivedreceived received Comment 165 105 145 Use link 125 to router 104. 165105 144 If link 125 is busy or down, use link 123 to router 102. 166 105146 Use link 156 to destination. 166 105 145 If link 156 is busy ordown, use link 125 to router 104. 165 104 142 Use link 155 todestination. 165 102 132 Use link 122 to router 104.

Routing information may include associations of policy values and portidentifiers (e.g., for network ports such as 165 and/or for router portssuch as 145). Generally a policy value includes any value that specifies(e.g., directly, or indirectly by identifying another specification)access permissions, desired quality of service, priority, connectiontype (e.g., connection oriented, connectionless) class of service,traffic class, or other transaction controls to be implemented before orduring routing. Policy values include any control values defined by aprotocol including the identification of the protocol (e.g., SCSI andversion number). For example, a Fibre Channel header includes a CS_CTLfield that describes a class of service having functional specificationsthat assure a particular quality of service.

When routing information is being prepared by edit maps process 302, anyrepresentation of a port may be used (e.g., a name from a name server,an index or pointer into a list of names, a mnemonic, an icon, a worldwide port name WWPN). Routing information may be entered by a systemoperator in any conventional manner including “point-and-click”,“drag-and-drop”, identification of a group of ports as equivalent (e.g.,ports 165 and 166 may be identified as functionally equivalent as tomember 115), or identification of a group of members of the network orof one or more subnetworks that are to be considered as a zone for acommon purpose such as specifying policy values.

In one implementation according to various aspects of the presentinvention, routing information between physical entities is developed byrouters 102-105 without user intervention according to methods performedby routers 102-105 that (a) identify port capabilities of all portscoupled to each port of a router; (b) advertise port identifiers toother routers via interswitch links; and (c) maintain routinginformation (e.g., further identification and advertising) when changesin port connections are detected. In such an implementation, the virtualports, virtual members, and virtual resources (with suitable policyvalues) might not be discovered by routers 102-105 and are developed byan administrating process with user input.

Routing information (e.g., paths database 303) may be stored andmaintained in a relational database. In one implementation, policyvalues are associated with group names, group names are associated withidentifiers of members, and zone names are associated with identifiersof resources (processes and devices). A zone name may be used todescribe a virtual member or a virtual resource. Further, group/zonetuples of group name, zone name, and policy values may be derived ormaintained in such a database. Still further, member/resource tuples ofmember identifier, resource identifier, and policy values may be derivedor maintained. In an alternate implementation, the derivation ofmember/resource tuples is accomplished by managing process 204 based onmaps received from administrating process 202.

Routers of network 101 may gather information useful for any portion ofadministrating process 202. Obtain reports process 304 may use anysuitable technique to obtain such information from routers 102-105. Forexample, obtain reports process 304 may poll routers of network 101 bysending a frame containing a command (e.g., a fabric control command orlink service request). Routers 102-105 may provide such information inany suitable form from which obtain reports process 304 formats one ormore entries in reports database 308. Obtain reports process 304 may,for example, use commands of the Simple Network Management Protocol(SNMP) to read any register or region of memory in a router 102-105. Forexample, routers 102-105 may provide lists of ports, image data, maps,and current configuration information from which administrating process202 may develop new, expanded, or revised paths.

A method for preparing a map according to various aspects of the presentinvention includes in any order: requesting member identifiers andresource identifiers from routers of the network; associating eachmember identifier to a group of members; associating each resourceidentifier to a zone of resources; associating a path and a policy valueto at least one of the group and the zone; determining port identifiersassociated with the path; and communicating the policy value to eachrouter of the network having at least one port identified to the path.In alternate implementations, the group and/or zone layer of indirectionmay be omitted so that path and policy values are associated directlywith member identifiers and resource identifiers.

A method for preparing a map that enables routing to virtual entities(e.g., virtual members, virtual resources) includes in any order: (a)providing a name for each of any number of virtual members and/orvirtual resources; (b) associating one or more portions of nonvirtualmembers and/or nonvirtual resources with each of the names; and (c)communicating each association to at least one router on each path usedto communicate between a nonvirtual member and either of a virtualmember or a virtual resource. The method may also include assigning anetwork port identifier to each virtual member or virtual resource. Whena resource is divisible into fungible units (e.g., identicallyfunctioning units of storage or processing such as sector addresses orobject references), the method may further include associating a unit ofa nonvirtual resource to the name or to a portion of a virtual resource.For example, a sector of a named virtual resource may be associated witha sector of a nonvirtual resource.

Policy values may include access control values (e.g., identifiers ofmembers or resources permitted to access other members or resources).Access controls may be associated with nonvirtual and virtual membersand resources.

A method for facilitating network traffic may include the steps of: (a)preparing a map to facilitate routing of frames referring to virtualentities; (b) providing policy values associated with virtual entities;and (c) communicating the identifiers of virtual entities only tomembers or resources permitted to access them. Communication of theidentifiers may be by selectively advertising.

A definition for implementing a database for use by edit paths process302 and storage of routing information in paths database 303 isdescribed in Table 3.

TABLE 3 FILE (or list) and associated fields or a record (or entry)Description GROUP/POLICY A named group may serve as a logical constructused to group_name describe a set of policy values. Group names may bepolicy_value associated with policy values in many-to-manyrelationships. In other words, several policy values may be associatedwith a group name and the group name serve as an indication of thecombination of policy values. Requesters (e.g., initiators) may beidentified to groups as opposed to zones. MEMBER/GROUP A member, asidentified by any suitable identifier (e.g., IP member_identifieraddress, or WWPN), may be associated with a group name to group_nameindicate that the policy values of the group are to be associated withall network traffic involving the member. Members may be associated withgroups in many-to-many relationships. An administration system operatormay designate a member_identifier (e.g., an unused name) as adesignation of one or more actual members. Such a member_identifier isherein called an identifier of a virtual member. ZONE/POLICY A namedzone may serve as a logical construct used to zone_name describe a setof policy values to be applied to resources and policy_value membersthat are part of the zone. Zone names may be associated with policyvalues in many-to-many relationships. Participants (e.g., targets) maybe identified to zones as opposed to groups. RESOURCE/ZONE A resource,as identified by any suitable identifier (e.g., IP resource_identifieraddress, WWPN; for a process, an object reference or zone_name referenceof the type used with CORBA), may be associated if virtual, theassociated with a zone name to indicate that the policy values of thenonvirtual resource identifier, zone are to be associated with allnetwork traffic involving and (if applicable) virtual unit to theresource. Resources may be associated with zones in nonvirtual unitcrossreferences many-to-many relationships. An administration system(e.g., page/sector table or object operator may designate aresource_identifier (e.g., an unused reference crossreferences) name) asa designation of one or more actual resources of one or more actualmembers. Such a member_identifier is herein called an identifier of avirtual resource. A virtual resource may be associated with an actual orvirtual member. A virtual member may have virtual resources. MEMBER/ZONEA member and all resources of the member (if any) may bemember_identifier associated with a zone as discussed above. zone_nameif virtual, associated nonvirtual member identifier PATH/PORT A namedpath may serve as a logical construct for developing path_name routinginformation. Typically two ports define the extremessource_port_identifier of a path: the port of a requester (e.g., asource) that is destination_port_identifier associated with a member orresource, and the port of a router_identifier participant (e.g., adestination) member or resource. An output_port_identifieradministrating process may have no knowledge of the routers and theiroutput ports that may be involved in alternate paths — leaving amanaging process in a router to obtain, integrate, and dynamicallymaintain such information, supplementing the definition of a path.Nevertheless, identifiers and ports may be associated to a path name tothe extent that an administrating process may suitably designatealternate routes or paths between groups, zones, members, and resources.MEMBER/PORT A port identifier may be a logical or physical reference toa member_identifier particular port. During member login to a port of aparticular router_identifier router, the member identifier and portidentifier may be port_identifier associated, for example, in a nameserver. A port identifier for a virtual member may be designated by anoperator of the administration subsystem. RESOURCE/PORT The identity ofa resource may be associated with a port of a resource_identifierparticular router, as discussed above. A port identifier for arouter_identifier virtual member may be designated by an operator of theport_identifier administration subsystem. GROUP/ZONE/PATH Theassociation of a path to a group, a zone, or both, provides group_namean association of policy values to the path. When policy zone_namevalues as defined for the group and the zone conflict, any path_namesuitable negotiation of policy values may occur to result inpolicy_values policy values to be used for the path. Integration ofpolicy values may follow predetermined hierarchical rules maintained byan administrating process (e.g., edit paths process 302) or by amanaging process. Negotiation may be accomplished dynamically during alogin sequence. Resulting policy values associated with a path form thebasis for routing tables. MAP A map may include policy values to beimplemented for any router_identifier or every router of the network. Asegmented map or an source_port_identifier overall map may be derivedfrom the records discussed destination_port_identifier above. In animplementation where routers develop routing policy_values informationwithout operator input, one or more of the fields crossreferences fordescribed here may be omitted. implementing routing for virtual entitiesROUTING TABLE A routing table for a particular router may be prepared asan source_port_identifier excerpt from a MAP. The operator of the systemdestination_port_identifier administration subsystem may determine whichrouters will policy_values use crossreferences for routing virtualentities. According to crossreferences for various aspects of thepresent invention, the readdressing of a implementing routing forvirtual frame that originally designated a virtual entity is entitiesaccomplished at any one router along a path; other routers along thatpath need not have access to crossreferences implementing routing forthat virtual entity. In one implementation, the burden of processingvirtual routing is distributed among routers of network 101. In animplementation where routers develop routing information withoutoperator input, one or more of the fields described here may be omitted.IMAGE An image may include information for routing, supervising,routing_table_entries and managing including data (e.g., constants,tables, routing_process_data configuration information), and programs(e.g., downloaded routing_process_programs subroutines for use by arouting processor to perform routing supervising_process_data of aparticular type of frame of a particular protocol issupervising_process_programs recognized by a parser). In a networkcapable of determining managing_process_data paths for actual membersand resources, the image may be managing_process_programs limited toinformation regarding routing of virtual

A port input/output process provides an application program interface(API) by which an application program may send and receive frames forcommunication (e.g., command, control, status, and data interchange)with other application programs, resources, and members of the network.For example, port I/O process 306 conducts all lower level protocols topermit administration process 202 to have access to information storedin routers and members of system 100. Port I/O process 306 provides anAPI to obtain reports process 304 and to manage link loads process 312.

Links are subject to traffic that consumes available capacity of thelink herein called a link load. The link load may be quantified ashaving frame rate, delays between frames, bursts of immediatelysucceeding frames, burst length, delays between bursts, and relatedderived quantities (e.g., maxima, minima, averages, counts, rates,variances) during a suitable duration of measurement or monitoring.Management of a link load at the level of system administration involvestransferring routing information from time to time to routers of network101; and, providing information to assist development of routinginformation. For example, manage link loads process 310 sends portionsof paths database 303 that apply to a particular router (e.g., rows 1-4of Table 2) as an update to the particular router (e.g., router 105) inany manner suitable for limiting the disruption of ongoing networkservices. Updates may occur when a router logs into the fabric and atany suitable time thereafter. Manage link loads process 310 may requestparticular reports from obtain reports process 304 or may make reportsfrom database 308 according to any conventional query.

The loads on various links over time and/or as related to one or moremembers constitute link utilization. Link utilization may be displayedin system 100 in any conventional aggregated or sorted manner. Forexample, display link utilization process 312 reads reports 308 andpresents link utilization to the system operator via a graphical userinterface. The system operator may analyze displays presented by displaylink utilization process 312 to determine that improved systemperformance may result if portions of paths database 303 are edited.Display link utilization process 312 may request particular reports fromobtain reports process 304 or may make reports from database 308according to any conventional query.

In each router of system 100 (e.g., 102), a managing process acceptspaths sent in frames to the router from an administrating process andprovides reports in frames to the administrating process. For example,managing process 204 accepts paths as sent by manage link loads process310 and provides reports from time to time to obtain reports process304. Due in part to the scalable architecture discussed above, eachrouter may receive updates from any administrating process and providereports as requested or automatically to any administrating process. Amanaging process, according to various aspects of the present invention,includes any process that performs one or more of the followingoperations: providing routing information to one or more supervisingprocesses; obtaining from one or more supervising processes informationfor reports as discussed above; governing operation of one or moresupervising processes to assure policy values are effected on aparticular link; serving as a proxy for one or more members in anycommunication (e.g., for virtualization); operating a cache to providean up to date redundancy of all or a portion of data stored at a member;and operating a cache to maintain a mirror storage resource as a copy ofanother storage resource. For example, managing process 204 includesport I/O process 402, LAN I/O process 404, manage configuration process406, map store 408, image store 409, obtain and supply reports process410, reports 412, load balance process 414, launch proxy for memberprocess 416, any number of proxy for member processes 418, proxy state420, cache agent process 422, cache 424, and mirror agent process 426.

Port I/O process 402 performs functions analogous to port I/O process306, discussed above.

LAN I/O process 404 provides an API for processes within router 102 tocommunicate via bus 210. Bus 210 may be of the type known as a localarea network (LAN), for example, including IP over IEEE 802.3 Ethernetfor supporting, among other functions, an interprocess protocol of thepromulgated by the Object Management Group as Common Object RequestBroker Architecture (CORBA).

Managing configuration for a router, according to various aspects of thepresent invention, includes establishing initial values and updates ofvalues stored in any memory device of the router. For example, manageconfiguration process 406 may receive configuration information (notshown) and routing information (e.g., paths) from administrating process202 by SNMP communication, in frames, or in file transfers (e.g.,comprising data in XML). Manage configuration process 406 determinesrouter specific routing and configuration information, and storesreceived and derived information in map store 408 and in image store409. Manage configuration process 406 determines configuration valuesthat may be suitably tailored for one or more supervising processes 206.Configuration information may be derived in accordance with theestablishment or termination of a proxy for a member, discussed below.Configuration information may also be derived in accordance with aresult of load balancing, discussed below.

Routing information (e.g., paths, and associations implementingvirtualization) may be received having references to logicalidentifiers. Mange configuration process 406 may refer to a name service(e.g., domain name service) to replace logical identifiers with physicalidentifiers and store results (e.g., maps) in map store 408. Maps in mapstore 408 may be used to develop routing information for particularrouting processes. Routing information particular to a routing processmay be combined with other data (configuration information, data, andprograms) to form an image for transfer to a routing process.

Image store 409 is organized for convenient access by manageconfiguration process 406. Manage configuration process 406 may accessimage store 409 for reading configuration information, forming a propermessage for the protocol on bus 210 (e.g., determining an address of asupervising process 206 for receipt of the configuration information);and for storing configuration information that may be reported bysupervising processes from time to time via bus 210. Manageconfiguration process 406 includes watchdog timers that notice when aconfiguration of a supervising process has changed, and when such aprocess is no longer responding. Manage configuration process 406 mayexecute a reset on any supervising process (or processor) in an attemptto re-establish proper operation of a supervising process (orprocessor). Image store 409 may contain a description of the state ofeach supervising process 206 managed by manage configuration process406.

Supervising processes are managed to coordinate operation of a router inan initial configuration, a power-on configuration (e.g., persistentfrom a recent power-off configuration), and an expanded configuration(e.g., additional ports and supervising processes added withoutdisrupting current routing functions). Configuration information to bestored in a memory device of the router includes codes (e.g., flags,identifications, controls, and interrupt settings) for command registers(herein called command/status registers (CSRs)), programs forinstruction stores (e.g., microcode for a state machine, nativeinstructions for a processor, or statements for an interpreter), andvariables and data for main memory (e.g., semiconductor and/or diskstorage for variables, tables, and related data for random access memoryor content addressable memory). These memory devices may be volatile ornonvolatile (herein generally called erasable programmable memory(EPM)). Consequently, manage configuration process 406 may conduct aseries of download operations via LAN I/O process 404 (in cooperationwith LAN I/O process 502) and may receive status and acknowledgementsfrom LAN I/O process 404.

A managing process may obtain reports from a routing process; and themanaging process may provide reports to an administrating process.Reports may be specified as to content and format by the administratingprocess and/or the managing process. According to various aspects of thepresent invention, communication of reports between all such processesutilizes network frames. The consuming process for any report mayrequest the report specifically each time it is desired, or specify asubscription for the report to be fulfilled without further interventionby the consuming process. The providing process may produce reports onlywhen requested (e.g., when polled), or may produce reports in responseto lapse of a timer or on the occurrence of an event (e.g., an abnormalcondition, or a condition requiring information and processing powerfrom a managing or administrating process). Any conventionalcommunication protocol may be used to implement the request/reply orsubscription mechanisms. A variety of protocols may be used for avariety of reports. For example, obtain and supply reports process 410sends requests via ports 214 (FIG. 2) conforming to SNMP to routingprocess 208 and routing process 208 sends replies via ports 214 toobtain and supply reports process 410 in conformance to SNMP. Moreparticularly, port I/O process 402 parses incoming frames and deliversframes identified as SNMP to obtain and supply reports process 410.Obtain and supply reports process 410 sends reports from managingprocess 204 via ports 201 to administrating process 202 via ports 107using network frames as discussed above.

According to various aspects of the present invention, a managingprocess may designate in a map a type of frame and an address recognizedby the managing process so that a routing process, operating accordingto the map, that receives a frame of the designated type will route theframe to the managing process. Generally, frames are of two types: thoseinvolving data transfer; and otherwise, those involving control and/orstatus. For example, manage configuration process 406 may specify in amap that control frames of particular protocols (e.g., controls forvirtual participants) are to be routed to a network port that isrecognized by the parser of port I/O process 402. Such a map, passed tomap 211 as discussed above, is used by routing process 208.

Part of a map may designate nonvirtual members (or resources) that areto be used when reference is made to virtual members (or resources).Each virtual member may accomplish the data processing and datacommunication functions of a member by obtaining the services of one ormore nonvirtual (i.e., actual) members, nonvirtual resources, orportions thereof. The designation of nonvirtual resources to a virtualmember may be specified by administrating process 202 and communicatedto managing process 204 as part of a map. Operations on a virtual member(e.g., by control frames or data frames) may be accomplished on aphysical device (e.g., member 110, or device 175 or 177), may beaccomplished on a logical device (e.g., member 115 corresponding toresources on subnetwork 170), or on another virtual device as long as anonvirtual device can be identified for the operations (e.g., nocircular references or undefined virtual identifiers).

Launch proxy for member process 416 includes any process that analyzesframes and prepares replies to accomplish any of the following: (a)establish a virtual member; (b) identify any or all existing virtualmembers; and (c) perform for a virtual member any action appropriate fora nonvirtual member (e.g., respond to any control frame). Communicationwith a virtual member, in accordance with various aspects of the presentinvention achieves the effect that the requesting member is unaware thatthe request was accomplished by a proxy as opposed to a nonvirtualmember. These determinations and replies may be accomplished usingprotocol analysis and communication techniques similar in some respectsto conventional parsers and port I/O processes suitably modified forlaunching and cooperating with one or more proxy processes. Launching aproxy includes maintaining a list of operating proxy processes,dedicating resources (e.g., memory in a managing processor) for use bythe proxy, determining an identifier for the proxy, updating routinginformation to enable communication with the proxy, and preparing toaccept status and error condition messages that may originate with or bea consequence of the proxy. For example, when port I/O process 402determines that a frame is a request to identify or to establish avirtual member, port I/O process passes the frame (or relatedinformation) to launch proxy process 416. Launch proxy process 416responds by identifying an existing proxy or launching a new proxy asdiscussed above.

A proxy process includes any process that receives frames in a firsttransaction and that prepares frames directed to a nonvirtual member orresource in a second transaction. The first and second transactions maybe in the same protocol or in different protocols. The first and secondtransactions may be separate in time or may overlap in time. Anonvirtual member or resource has state according to the protocol usedto communicate with the nonvirtual member or resource. A proxy processmakes virtual state visible to the user of the virtual member orresource (e.g., in response to a control frame). For example, each proxyprocess 418 operates as if it were a nonvirtual member having stateaccording to the protocol used by the user of the virtual member. Thestate of the nonvirtual member and the virtual state as made visible bythe proxy to the user of the virtual member may differ. When theprotocol used with the nonvirtual member is identical to the protocolused with the virtual member, the respective states may correspond. Evenso, these states may differ, for example, because of temporaldifferences between the conduct of the first transaction and the secondtransaction.

Proxy state includes any data structure for maintaining the state of thevirtual member (or resource) and the state of the correspondingnonvirtual member (or resource). For example, proxy state 420 includes aconventional data base stored in any suitable memory (e.g., acombination of semiconductor and disk memory devices). Proxy state 420may comprise any suitable organization of records with fields asdescribed in Table 3A.

TABLE 3A Field Description on-line Describes whether the device (amember or a resource) is available for login to the network; logging-inDescribes whether the device is currently participating in a log-inscenario; device Describes whether the device is currently assigned avalid port available which may be specified in a transaction; accessibleDescribes for a storage device what portions of the storage pages deviceare ready (or will be ready) for immediate access due in part to thestructure (e.g., a portion of the storage medium proximate to theread/write head) and operation (e.g., seek times to other cylinders, orto other portions of streaming tape) of the storage device.

A cache agent process includes any process that maintains data in arelatively faster access memory so that reference to a relatively sloweraccess memory may be avoided. The capacity of the faster access memoryis generally subject to limitations on the amount, type, or organizationof data stored therein. A cache agent process receives requests for databearing suitable identification of the desired data, examines the cachefirst and if the desired data is not there, obtains the data andpossibly stores the data in the cache to facilitate future reference toat least part of the same data. If the data is in the cache, the cacheagent process provides the data in response to the request and may notethat the data has been accessed. The cache agent may determine whetherto retain data in the cache in response to notations as to its havingbeen accessed (e.g., time of last access, total number of accesses in aperiod of time, and/or identity of the requester for which access wasmade or destination to which the data was provided). For example, cacheagent 422 receives requests from port I/O process 402, performs thefunctions of a cache agent as discussed above by accessing cache 424,and directs port I/O process 402 to reply with data as requested to besent to the requester.

A mirror agent process includes any process that maintains more than onecopy of particular data. A second copy of data (also called the mirror)when read at any time must provide the same result as reading theprimary copy of data (the data that is being mirrored). The primary datais expected to be subject to change by being written. To properly mirrorprimary data that has been written, the write to the second copy must bemade prior to a read of that portion of the second copy that would beaffected by the write. According to various aspects of the presentinvention, a mirror agent process may prepare and maintain the secondcopy without initially preparing a complete copy of the primary data. Inother words, the second copy may at any instant of time (a) be empty,(b) contain primarily or exclusively a copy of the data that has beenwritten to the primary copy, or (c) contain primarily or exclusively acopy of the data that has been read from the primary copy. By delayingcopying unused portions of the primary copy to the second copy, networktraffic may be more effectively used for other network functions. Forexample, mirror agent process 426 receives requests from port I/Oprocess 402 to maintain one or more copies of identified data, performsthe functions of a mirror agent as discussed above by accessing cache424, and directs port I/O process 402 to perform reads of the primarycopy and writes to the second copy to maintain the second copy asdiscussed above.

A cache includes any data structure for facilitating access to data asdiscussed above. For example, cache 424 includes a conventional database stored in any suitable memory (e.g., a combination of semiconductorand disk memory devices).

Each routing process may communicate with a managing process 204 or itscomponents (406, 414, 410, 416, 418, 422, 426) using one or more networkport identifiers (e.g., destination addresses). Such a network portidentifier may be a predefined address, an address reserved to therouter, a world wide port name, or a so-called well known address.Network port identifiers may be used by routing processes and byadministrating processes to communicate with managing processes. Arouting process generally communicates primarily or exclusively with themanaging process in the same router as the routing process.Communicating by use of a network port identifier of network 101 is alsocalled “in-band” communication. By contrast, networks 210 and 212, forexample, do not represent “in-band” communication.

A supervising process, according to various aspects of the presentinvention communicates with one or more managing processes 204 via LAN210; and, communicates with any number of routing processes 208 via atleast one of bus 212 and shared memory 211 as discussed above. Suchcommunication maintains a current map 211 for use by each routingprocess and accomplishes link services for the links maintained by eachrouting process. For example, supervising process 206 includes LAN I/Oprocess 502, image store 503, update images process 504, get linkservice request 506, put link service reply 507, control fabric process508, log 510, serve names process 512, namestore 514, broadcast process516, and group store 518.

Supervising process 206 cooperates with manage configuration process 406to receive routing information. LAN I/O process 404 provides routinginformation according to a protocol followed also by LAN I/O process 502for receipt and acknowledgement of routing information. LAN I/O process502 may also provide indications that supervising process 206 isoperating properly via the cooperation of LAN I/O processes 502 and 404.LAN I/O process 502 analyzes routing information that has been receivedand stores routing information in image store 503.

Image store 503 is organized for convenient access by LAN I/O process502 and update images process 504. Image store may include codes,programs for instruction stores, variables, data, and routinginformation, as discussed above with reference to image store 408.Update images process 504 may access image store 503 for readingconfiguration information, forming a proper message for the protocol onbus 212 (e.g., determining an address of a routing processor for receiptof the configuration information), and for storing configurationinformation that may be reported by routing processes from time to timevia bus 212. Update images process 504 includes watchdog timers thatnotice when a configuration of a routing process has changed, and whensuch a process is no longer responding. Update images process 504 mayexecute a reset on any routing process (or processor) in an attempt tore-establish proper operation of a routing process (or processor). Imagestore 503 may contain a description of the state of each routing processmanaged by update routing memory process 406. Update images process 504reads image data from image store 503 and stores image data for accessby routing processes as discussed above with reference to map 211. Namestore 514, map 211, and group store 518 may receive initial values andbe updated from image store 503. For example, identifiers and policyvalues (including access control values) for virtual members and virtualresources may be stored in name store 514 to be advertised or providedon request by serve names process 512 (e.g., in one implementation onlypermitted access is facilitated by selectively providing virtualidentifiers in accordance with access control values). Image data mayinclude data to be referenced by, and instructions to be performed byone or more routing processes. Update images process 504 monitorsrouting processes 208 in any conventional manner and initializes andupdates image data in map 211 at any suitable time or interval.

A link service request is a request sent by a member or resource ofnetwork 101 that can be accomplished with reference to data maintainedby a router. Generally, a link service request is completed with a linkservice reply. Requests for data transfer between members are generallynot considered link service requests. Link service requests aregenerally defined by a protocol of network 101. When router 102 supportsmore than one protocol, one or more supervising processes may coexist inrouter 102, for example, one process for each protocol. For example, getlink service request process 506 and put link service request 507perform conventional interprocess communication between supervisingprocess 206 and one or more routing processes 208. Link service requestsmay be processed in any conventional manner. For example, get linkservice request process 506 distinguishes fabric control requests, nameservice requests, and broadcast requests and routes respective requeststo control fabric process 508, serve names process 512, and broadcastprocess 516. Each of these processes prepares a suitable reply for useby put link service reply process 507. Put link service reply process507 provides the reply to the routing process that made the request.Table 4 describes representative link service requests and processing.All of the processes described in Table 4 may invoke action by put linkservice reply process 507 to generate a suitable reply to each linkservice request. A reply may describe result conditions or errorconditions concerning the link service request. A proxy process 418performed by managing process 204 may initiate any control frame for avirtual member or resource (e.g., initiate a link service request,log-in a virtual member, or designate a quality of service for a virtualresource).

TABLE 4 Link Service Request/Reply Description of Processing Abort atransaction Control fabric process 508 revises log 510 and notifiesstatistics gathering processes (if any) so that the transactionidentified in the link service request is interrupted without completingthe request of that transaction. Remove a connection Control fabricprocess 508 revises log 510 and notifies statistics gathering process(if any) so that the path identified in the link service request isinterrupted. Such a path may be a dedicated path. Ports involved alongthe path (or paths) are freed for general use. Log-in Control fabricprocess 508 revises log 510 and may notify routing process 208 to reportcurrent members of the network to administrating process 202. Servenames process 512 revises name store 514 with a new or unused name forthe device. A port is identified to the device that desires to become amember of the network via the port. Log-out Control fabric process 508revises log 510 and may notify routing process 208 to report currentmembers of the network to administrating process 202. Serve namesprocess 512 revises name store 514 to disassociate the name from thedevice that was a member. A port is disassociated from the device toremove the member from the network. Implement a quality of Controlfabric process 508 revises log 510 and map 211 from which servicerouting process provides the quality of service specified in the linkservice request. Implement a buffer Control fabric process 508 revisesmap 211 from which routing process credit or grant 208 dedicates orfrees buffer space for frame handling. When managing process 204provides a proxy process implicated in this link service request,control fabric process 508 cooperates with managing process 204 toimplement the requested credit or grant. Implement a group Broadcastprocess 516 revises group store 518 and map 211 from address whichrouting process 208 operates to broadcast or multicast frames to morethan one destination.

A routing process generally routes frames by analyzing each framereceived from a port, selecting suitable routing information, andproviding at least the received payload in the same or a correspondingframe to an output port. A routing process provides access to membersand resources as requested by a member (or resource) that has suitablepermission (e.g., via an access control list), provides quality ofservice according to suitable policy values, and maintains transactionswith physical, logical, and virtual entities. A routing process alsoobtains and reports statistics. For example, routing process 208includes statistics store 601, report status and errors process 602,pass link service request process 603, supervisor queue 604, field linkservice reply process 606, route frame to fabric process 608, manageoutput queues process 610, manage egress queues 612, egress buffer 614,ingress buffer 616, flow process 618, pass to proxy process 620, routingtable 622, subflow process 624, context table 626, virtual flow process628, virtual context table 630, page table 632, sector table 634,virtual port identifier table 636.

Routing process 208 provides reports 214, 201 to managing process 204.Information reported describes traffic through router 102. Each routingprocess 208 may accumulate counts of the quantity of frames satisfying avariety of criteria. These counts and data derived with reference to thecounts is stored in statistics store 601. Counts may accumulate over aperiod of time fixed, specified by supervising process 206, ordynamically determined by routing process 208. Data may include one ormore of the following computations: average, ratio, net change, rate ofchange, variance, standard deviation, and binary results from acomparison of a current value of data to a threshold that may be fixed,specified by supervising process 206, or dynamically determined byrouting process 208. Access to statistics may be indexed in anyconventional manner.

The subject of a count or derivative may be limited to a physical port,logical port, virtual port, flow, subflow, virtual flow, member, orresource identified for example, by analysis of one or more fields in aframe (e.g., pattern matching by a parser circuit).

Report status and errors process 602 reads statistics 601 or determinesstatus, configuration, or error conditions of routing process 208 andprepares a suitable report. Report preparation may be automatic (e.g.,on occurrence of an error or lapse of a reporting time period) or polled(e.g., in response to a request 201, 214 from managing process 204).

Pass link service request process 603 formats information recalled fromstatistics store 601 or received from report status and errors process602. Pass link service request process 603 also formats informationreceived by flow process 618 so that any portion of a link servicerequest frame may be provided to supervising process 206. Pass linkservice request process 603 stores the formatted information insupervisor queue 604.

Supervisor queue 604 serves as a buffer between pass link servicerequest process 603 and field link service reply process 606.Supervising process 206 may access supervisor queue 604 in each ofseveral routing processes as described above with reference to bus 212.By buffering link service requests, (a) supervising process 206 mayimplement priorities for the execution of link service requests (in anorder other than as requested) and processing of reports (in an orderother than as polled or as available); (b) pass link service requestprocess 603 may specify and revise priorities among outstanding items inqueue 604, (c) supervising process 206 may delay processing ofparticular link service requests or reports, (d) results of processingby supervising process may be noted in queue 604, and (e) field linkservice reply process 606 may act on replies from queue 604 in any orderand at any suitable intervals, allowing route frame to fabric process608 to implemented priorities without loss of data.

Field link service reply process 606 reads replies from supervisor queue604 (entered into the queue in response to a link service request orreport as discussed above). Reading may be responsive to thresholds toavoid backlog in queue 604, may be upon lapse of a time period (fixed,specified, or determined as discussed above), or may be upon requestfrom route frame to fabric process 608. Field link service reply process606 prepares a suitable link service reply frame that may include dataread from or derived with reference to queue 604 and passes the frame toroute frame to fabric process 608.

A fabric is a mechanism that provides access to data among numeroussource processes and destination processes. In one implementation afabric comprises a multiported memory allowing any number of sourceprocesses to write into the memory and any number of destinationprocesses to read from the memory. In another implementation the fabriccomprises a network that makes a copy of data from a source buffer intoone or more destination buffers. Source and destination buffers may thenbe implemented as memories with much simpler access functions; a sourcebuffer is read by the network and written by one source process; and adestination buffer is written by the network and read by one destinationprocess. Network processes at each of several destination buffers mayimplement multicasting or broadcasting by storing a copy from amulticast or broadcast source that is made available to all destinationnetwork processes. For example fabric 213 provides communication amongany suitable number of routing processes 208 each having respectiveprocesses 610 and 612. Fabric 213 may implement communication with anycombination of multiport memory and network technology as discussedabove.

Route frame to fabric process 608 reads the destination port identifierassociated with data received from any process 606, 618, 620, 624, or628 and passes the data to manage output queues process 610 with adesignation of one or more output queues. From the perspective of fabric213, output queues of a first routing processor's ingress buffer, mayserve as source buffers to be transferred to another routing processor'segress buffer serving as a destination buffer, as discussed above. Routeframe to fabric process 608 may read data so as to implement servicepriorities among processes 606, 618, 620, 624, and 628. The priority ofdata read may be determined by route frame to fabric process 608 inaccordance with an identifier of the requesting member, resource, orport (e.g., a source identification), an identifier of a participatingmember, resource, or port (e.g., a destination identification), whichprocess 606, 618, 620, 624, or 628 provided the data, statistics fromstatistics store 601 related to a characteristic of the data or relatedto a process 606, 618, 620, 624, or 628 in a period of time (fixed,specified, or determined as discussed above), a priority associated withthe data by the process 606, 618, 620, 624, or 628, or a policy valueassociated with the data by process 606, 618, 620, 624, or 628. Routeframe to fabric process 608 may format the data as the payload of aframe according to framing conventions used for fabric 213 and/orframing conventions used for network 101.

According to various aspects of the present invention, data received forrouting by process 608 includes policy values from which a suitableoutput queue may be determined by route frame to fabric process 608.

Manage output queues process 610 receives frames from route frame tofabric process 608 and transfers each frame to fabric 213. Manage outputqueues process 610 may maintain a plurality of output queues, eachoutput queue corresponding to a physical port of router 102 (e.g., aport connected to a member that issues frames into router 102 forrouting). Manage output queues process 610 may arbitrate among queues toefficiently access fabric 213, or to implement a policy associated witha particular queue or a policy associated with a particular frame. Forexample, when fabric 213 includes a network as discussed above, manageoutput queues process 610 may add fabric network framing to hide network101 framing in the payload of a fabric network frame.

Preferably, frames having different associated policy values (e.g.,different traffic class or different class of service) are enqueued intoseparate queues, subject to queue servicing rules implemented by manageoutput frames process 610. Further, frames may be enqueued according tosource identification, destination identification, and policy values(e.g., one queue for every combination of physical input portidentifier, traffic class value, and physical output port identifier).

Manage egress queues process 612 receives (or recalls) frames fromfabric 213 and transfers each frame to one or more egress buffers 612,each egress buffer may correspond to a physical output port of router102 (e.g., a port connected to a member that consumes frames from router102 after routing). Manage egress queues process 612 may maintain aplurality of egress queues to effect arbitrated access to one or moreegress buffers and/or to effect flow control back toward fabric 213.Arbitration and/or flow control may implement a policy value associatedwith a particular egress buffer or a policy value associated with aparticular frame. Data from fabric 213 may be reformatted by manageegress queues process 612 to comply with signaling and framing standardsof network 101. For example, when fabric 213 includes a network asdiscussed above, manage egress queues process 612 may strip fabricnetwork framing to expose network 101 framing.

Egress buffer 614 supplies frames to network 101. Egress buffer 614 mayinclude a large number of queues for storing frames that awaittransmission onto network 101. While in storage, a frame in a queue maybe revised, for example, to accomplish support for virtualization. Whena frame carries a payload from a nonvirtual transaction that is to bedelivered to a participant of a virtual transaction, manage egressqueues process 612 may: (a) parse the frame from fabric 213; (b)determine that modification is desirable; (c) recall at least a virtualdestination port identifier from virtual context table 630; and (d)modify the frame's destination port identifier in accordance with thevirtual destination port identifier before transmitting the payload tonetwork 101.

Ingress buffer 616 receives frames from network 101. Ingress buffer 616may include a large number of queues for storing frames that awaittransmission onto fabric 213. While in storage, a frame in a queue maybe revised, for example, to accomplish support for virtualization.

Flow process 618 reads frames from ingress buffer 616, parses,classifies, and processes each frame as described in Table 5.

TABLE 5 Frame Contents Description of Processing Link service requestWhen parsed results indicate the frame is a link service request, flowprocess 618 passes any or all of the frame to pass link service requestprocess 603. Indications that a frame is a link service request includethe destination address portion of the frame (e.g. an address reservedfor link service requests according to a protocol of network 101), avalue describing a type of frame, and/or a value describing a protocolto which the frame is compliant. Frame for notice to a When parsedresults indicate the frame is of a type to be supplied to a proxy or foraction by proxy, flow process 618 passes any or all of the frame to passto proxy a proxy process 620. Such a frame may be a control frame ordata frame regarding a transaction involving a virtual member orresource. Such a frame may notify the proxy, effect the state of aproxy, or trigger suitable action by the proxy. Indications that theframe is of a type to be supplied to a proxy include the destinationport identifier portion of the frame (e.g. a network address reservedfor a proxy according to a protocol of network 101, or any fields of theframe identified for processing by a proxy, for example, by anassociated flag obtained from routing table 622 accessed in accordancewith a portion of the frame), a value describing a type of frame, and/ora value describing a protocol to which the frame is compliant. Frameunrelated to a When parsed results indicate that the frame is unrelatedto a subflow or subflow or virtual flow a virtual flow, flow process 618passes any or all of the frame to route frame to fabric process 608.Indications that a frame may be unrelated to a subflow or virtual flowinclude an associated flag obtained from routing table 622 accessed inaccordance with a portion of the frame, or simply a value of adestination port identifier field of the frame. Frame related to a Whenparsed results indicate that the frame is related to a subflow, flowsubflow process 618 passes any or all of the frame to subflow process624. Indications that a frame is related to a subflow include anassociated flag obtained from routing table 622 accessed in accordancewith a portion of the frame. Frame related to a When parsed resultsindicate that the frame is related to a virtual flow, virtual flow flowprocess 618 passes any or all of the frame to virtual flow process 628.Indications that a frame is related to a virtual flow include anassociated flag obtained from routing table 622 accessed in accordancewith a portion of the frame, and/or a value of a destination portidentifier field of the frame. None of the above Flow process 618 maydrop the frame by freeing the ingress buffer space allocated to theframe. Flow process 618 may raise a countable statistic or an errorcondition in concert with dropping a frame. Flow process 618 may passany portion of the frame to report status and errors process 602 tofacilitate rectifying the error condition or avoiding future errorconditions.

Any of the references made to routing information discussed in Table 5may provide one or more policy values for output queue selection asdiscussed above.

Pass to proxy process 620 may associate the data (corresponding to aframe) received from flow process 618 with an identifier of a particularproxy for member process 418 and revise the frame accordingly. Anidentifier may be selected from a range of network port addresses notused by router 102 yet reserved to router 102 by a protocol of network101 (e.g., well known addresses). The identifier may further include anobject reference. Pass to proxy process 620 then passes the data and theproxy identifier to route frame to fabric process 608. In oneimplementation, when a requester directs a control frame or a data frameto a virtual entity, the frame includes a destination port identifierthat identifies the proxy that acts for the virtual entity. Toaccomplish passing to the proxy, pass to proxy process 620 may routesuch a frame without revision.

Routing table 622 includes cross reference information received from map211 and information determined by flow process 618. For example, routinginformation as discussed above may include a tuple (e.g., anassociation) of source identifier/destination identifier that may beused to obtain routing information for egress (e.g., an identifier of aqueue, a logical router port identifier, or a physical router portidentifier). Such a tuple is herein called a flow; and, a row of therouting table is herein called a flow entry. Generally, informationregarding one flow may be organized in one row of routing table 622.Where more than one row is made necessary by the quantity of informationor for representing many-to-one relationships, a portion of a row (e.g.,a flow identifier) may be used in a subsequent access of the routingtable. The subsequent access is herein called a subflow. Subflow entriesmay be used to describe resources on a subnetwork of a member asdiscussed above.

The routing information for egress recalled from routing table 622 maycorrespond to an output queue 610, a fabric network address, an egressqueue 612, and/or an egress buffer 614. Particular advantages arerealized by identifying each of the above to the same physical portidentifier so that the destination port identifier is sufficient todirect the frame out of the appropriate physical port of router 102. Thesame tuple may be used to obtain (e.g., simultaneously with the physicalport identifier for egress) one or more policy values used to implementpolicies as discussed above.

Information determined by flow process 618 may include an identifier ofa resource from a request frame. For example, when a request frameincludes a destination port identifier of a member, a transactionidentifier, and a resource identifier (the resource being on asubnetwork of the member) subsequent frames from the requesting memberor from the resource that accomplish data communication may omit theresource identifier relying on the destination member identifier and/orthe transaction identifier for routing. In such a case flow process 618may determine that the frame is a request conforming to a protocol thatmakes such an omission and store in routing table 622, context table626, or virtual context table 630 the resource identifier in associationwith the transaction identifier and/or in association with thedestination port identifier for future reference.

Subflow process 624 generally receives from flow process 618 dataregarding a frame addressed to a member and a resource on a subnetworkof the member. Subflow process 624 associates the data with a routerport identifier. Subflow process 624 may obtain the router portidentifier and policy values from routing table 622 as a flow lookupdiscussed above. Subflow process 624 may read a subflow (e.g., perform asubflow lookup) from routing table 622 accessed in accordance with aportion of the data and/or information recalled from the flow lookup.Subflow process 624 may further read context table 626 as directed byinformation recalled in the flow lookup and/or the subflow lookup and/orby a portion of the data. Subflow process then applies policiesindicated by policy values that may be associated with the flow and/orthe subflow entries in routing table 622 and/or associated with theresource entry in context table 626. Subflow process 624 then passes thedata, the router port identifier, and policy values, to route frame tofabric process 608.

When a transaction is begun involving one or more virtual devices(herein called a virtual transaction) routing process 208 identifies aframe that signals the beginning of the virtual transaction, and inresponse to that frame and in accordance with the protocol identified tothe virtual transaction, performs the remainder of the virtualtransaction in concert with beginning and performing a correspondingtransaction with a physical member and/or device (herein called anonvirtual transaction). The protocol used in the nonvirtual transactionmay differ from the protocol used in the virtual transaction. In otherwords, there may be no one-to-one correspondence between frames (e.g.,frames for inquiry, data transfer, reply, status, and error conditions)of the virtual transaction and frames of one or more nonvirtualtransactions that implement the virtual transaction on nonphysicalmembers and/or nonvirtual resources. Policies implemented for thevirtual transaction may differ from policies implemented for thenonvirtual transaction, for example, to assure meeting a policyassociated with the virtual transaction.

Virtual flow process 628 receives from flow process 618 datacorresponding to a frame of a virtual transaction (e.g., addressed to avirtual member and/or virtual resource). Virtual flow process 628associates the data with a router port identifier and prepares data fora frame of a nonvirtual transaction (e.g., addressed to a nonvirtualmember and/or a nonvirtual resource). Virtual flow process 628 mayobtain the router port identifier and policy values from routing table622 as discussed above as a flow lookup using a tuple of sourceidentifier/virtual destination identifier. Virtual flow process 622 mayread context table 626 as discussed above as a subflow lookup using thesame tuple as for the flow lookup accompanied by a portion of theresults (e.g., flow identifier) of the flow lookup and/or data fromprocess 618.

A nonvirtual resource may have a state different from the state of thecorresponding virtual resource. The state of a virtual resource may betracked by a proxy as discussed above with reference to proxy state 420.For example, support for a virtual storage resource may allow read/writeaccess in a manner unsuited to efficient operation of a physicalresource (e.g., contiguous sectors in reverse order of cylinder spin) soas to satisfy particular efficiencies realized by a process of therequesting member. An implementation of such a virtual storage resourcemay include caching and buffering as discussed above with reference tocache agent 422. Further, a virtual storage resource may be mapped(e.g., on a sector basis) to any mix of nonvirtual devices and portionsof nonvirtual devices. A virtual storage resource may be accessed as aconventional block device having virtual cylinders comprising virtualpages, and virtual pages comprising virtual sectors.

Virtual flow process 628 may use identifiers recalled from the flowlookup, the subflow lookup, and/or the context table 626 to determine anonvirtual resource identifier; and then refer to page table 632 andsector table 634 to obtain virtual to nonvirtual cross references fromwhich a nonvirtual page and sector (e.g., a nonvirtual block) may beidentified. After the nonvirtual destination port and nonvirtual blockare determined, virtual flow process may perform a logical flow lookupand possibly a logical subflow lookup to obtain a router output portidentifier, nonvirtual resource identifier, and policies to implement.In an alternate implementation, the router output port identifier,nonvirtual resource identifier, and policy values are obtained directlywith the initial flow and subflow lookups without a logical flow orlogical subflow lookup.

Particular advantages are realized by locating logical to physical andvirtual to nonvirtual cross reference information in tables that may beaccessed by multiple routing processes (e.g., shared memory). Port table636 may be stored in shared memory indexed by logical port identifier toprovide a corresponding physical port identifier (more than one may beprovided for broadcast and multicast applications). A logical portidentifier may correspond to routing information provided by anadministration process as discussed above (e.g., a group name, zonename, path name, or suitable reserved label). Virtual context table 630may be stored in shared memory indexed by an identifier of the virtualmember, virtual resource, and/or virtual transaction to provide acorresponding nonvirtual transaction identifier. In an alternate virtualflow process implementation, the virtual flow process obtains the routeroutput port identifier (e.g., a logical to physical lookup) and may alsoobtain policy values by accessing either port table 636 or virtualcontext table 630.

Virtual flow process 628 identifies data to route frame to fabricprocess 608 for use in one or more frames for one or more nonvirtualtransactions that implement the virtual transaction indicated by framesreceived by flow process 618. Data may include the router output portidentifier, nonvirtual resource identifier, nonvirtual block, nonvirtualtransaction identifier, and policy values.

A method for routing frames according to various aspects of the presentinvention includes any method that includes one or more of thefollowing: (a) implementing different policies for each of differentresources that may share a common member identifier, (b) implementingone or more nonvirtual transactions to accomplish the intent of avirtual transaction; (c) obtaining nonvirtual block identificationcorresponding to virtual block identification; (d) arbitrating amongqueues on the basis of a grant pool for each of a plurality of servicetypes or traffic classes, and (e) implementing a stall for one ofseveral resources that share a common member identifier or resourceidentifier. For example, a method 700 of FIGS. 7-10 that is performed byany router 102-105 as described above and may be performed by anyrouting process 208 proceeds as follows.

To process a flow, a frame is received comprising indicia of a desiredflow (702). The desired flow may be indicated by any combination of asource identifier, a destination identifier, and a protocol. Indicia ofthe flow are used as an index (704) to obtain flags, policy values, andan output queue identifier, all from one or more tables (e.g., eachtable may be a data structure, a record of a database, or a set of datastructures or records of a database). The flags are then used (706) todetermine which of five processing scenarios should apply to the subjectframe.

Use as an index includes use in an exact match search and use in amaximal match search. Searching may be facilitated by contentaddressable memory circuitry that receives the index (e.g., a tag havingdata and ternary designations: must match, must not match, don't care)and provides flags indicating the extent of the match. When more thanone match is found, use of the maximal match is preferred. A match maybe better (more maximal) than another match when more fields of the tagmatch, when higher priority fields of the tag match, or a weightedcombination of component fields matches. When tag fields are arranged bypriority (or weight), a longest match (e.g., greatest number ofcontiguous fields or bits) may provide a maximal match. A field valuemay indicate a wild card accepting any result as a match.

A transaction may include several frames to be routed. In the followingdiscussion, routing frames of a transaction is accomplished by routingall frames of a transaction primarily for control as control frames andall frames of a transaction primarily for data transfer as data frames.

If the flags indicate the frame is a link service request, the frame ispassed (708) to a supervising process that accomplishes the intent ofthe link service request as discussed above. As a consequence ofprocessing the link service request, data may be provided by thesupervising process for a frame to be placed (720) in the output queueidentified previously (704).

If the flags indicate a type-A nonvirtual frame, one or more policiesare applied (718) to effect a quality of service and the frame is placed(720) in the output queue identified previously (704).

If the flags indicate a type-B subnetwork transaction, a resourceidentifier and policy values associated with the resource identifier areobtained (712) first by parsing the frame according to the protocol todetermined the resource identifier of the subnetwork of the destinationand second by using the resource identifier in a subflow lookup to getpolicy values that have been associated to the destination portidentifier and the resource identifier. Then one or more policies areapplied (718) to effect a quality of service, and the frame is placed(720) in the output queue identified previously (704).

If the flags indicate a type-C virtual data frame, the identifiersdetermined by prior parsing (702) are taken as virtual source identifierand virtual destination identifier. Data for a nonvirtual transactionframe is obtained (714) by further parsing the received frame (702)according to the protocol to determine a virtual resource identifier andvirtual block description. The virtual resource identifier is translatedby reference to one or more cross-reference tables (e.g., tables of theform discussed above at 704) to a nonvirtual resource identifier. Thevirtual block description is translated by reference to one or morecross-reference tables (e.g., tables of the form discussed above at 704)to a nonvirtual block description. Processing as discussed for type-Bframes may be accomplished for the nonvirtual destination portidentifier and the nonvirtual resource identifier; or, policiesidentified with the nonvirtual destination identifier and nonvirtualresource identifier are applied (718) and a frame comprising thenonvirtual resource identifier and the nonvirtual block description isplaced (720) in the output queue identified to the nonvirtualdestination port identifier (704).

If the flags indicate a type-D virtual control frame, the frame isidentified (716) to be routed to a suitable proxy, the frame is placed(720) in the output queue associated with a managing process 204 orproxy process 418 (704).

After a suitable frame has been placed in an output queue, processingcontinues with the next frame (702).

To implement a policy according to various aspects of the presentinvention, data particular to a transaction is maintained up to date.Such data may include the state of a resource, proxy state, and/orcross-reference information for determining a nonvirtual transaction forimplementing a virtual transaction. For example, when administratingprocess 202 defines a new or revised virtual member or virtual resource,managing process 204 may launch a new proxy process 418, and managingprocess 204 in cooperation with supervising process 206 may update map211 for use by all routing processes 208. Proxy state is consequentlyupdated. When a transaction is completed normally or terminatedabnormally, data particular to the transaction (e.g., a saved resourceidentifier, or statistics) may be discarded and processing resourcesthat may have been allocated are freed. Routing process 208 maintainstransaction data (802) by cooperating with a supervising process forshared access to map 211.

Queue controls (804) and arbiter controls (806) are set in accordancewith policy values. Queue controls may designate priorities amongcompeting queues, flow control strategies and thresholds for each queue(e.g., actions to take when a queue is getting full or getting empty),and/or effect a stall on a queue preventing further input (e.g.,allowing an input queue to empty) or preventing further output (e.g.,allowing an output queue to fill). Arbiter controls may designate flowcontrol strategies and thresholds for each of a group of queues of thesame priority (e.g., same traffic class). Queue controls and arbitercontrols may be set by register transfer instructions when queue controland arbitration are effected by logic circuits. Application of a policymay include accumulating (808) statistics related to frames routedand/or queue and arbiter operations for use by a managing oradministrating process as discussed above.

According to various aspects of the present invention, multiple copiesof information from a frame are avoided to avoid the time memory spaceconsumed by making a copy. The one copy of frame data may persist in aningress buffer until all reference to it has been accomplished (e.g., acorresponding frame is transferred to the fabric or the frame isdropped). In the discussion above regarding passing a frame or data of aframe among processes, the data that is passed may be merely a pointerto the ingress buffer where frame data can be read indirectly (via thepointer), a handle to context where pointers and simple values arestored, or a pointer to a row of a table where a translation may beobtained.

Placing a frame in an output queue may be accomplished in a manner thatimplements a policy. The result of such placement in a non-blockingrouter is that the frame is eventually transmitted out of the router inaccordance with a priority. The entry in the queue may be a reference toframe data in an ingress buffer as discussed above, or a handle to acontext having pointers and simple values as discussed above. Each queuemay be a linked list of ingress buffer contents.

Placing an item into such a queue (enqueueing) may include inserting anitem into a linked list (e.g., storing revised values of pointers). Apolicy may affect any of several steps in routing a frame. Routing mayinclude, for example, enqueueing a frame for transmission onto fabric213 by making reference in a suitable first queue to the frame as it isstored in ingress buffer 616; servicing the first queue by a firstarbiter for transmitting the frame onto fabric 213; receiving the frame(e.g., essentially the payload) from fabric 213 into egress buffer 614;enqueueing the received frame by making reference in a suitable secondqueue to the frame as it is stored in egress buffer 614; and servicingthe second queue by a second arbiter for transmitting the frame tonetwork 101. The first and the second arbiters may use the same ordifferent arbitration techniques.

The amount of space available for frames in a buffer used for a queuemay be managed by several protocols of fabric 213 and network 101 (e.g.,backpressure logic or techniques of the type used in Fibre Channel)wherein requests for buffer space are sent to a receiving port andgranted with the result that an integral number of credits correspondingto reserved buffer space are received by the requesting port. Buffercontents may be later transferred to another buffer or region of memorywhere available space must be requested in advance in a similar manner(e.g., a buffer dedicated to a particular resource at the end of thesegment, or a number of buffers (e.g., end-to-end) along multiplesegments (e.g., hops) of a communication path through network 101. Asused herein, a grant or grant pool refers to a buffer space allocationmechanism at any level of communication protocol (e.g., a credit orallowance in addition to a credit). Grants may be associated with aresource, a segment, a port, an ingress or egress buffer, or a fabricchannel.

Any conventional arbitration may be used for arbiters as discussedabove. Particular advantages are realized according to various aspectsof the present invention by implementing queues with timers. Each timermay facilitate minimal fractional bandwidth for one or more queues.During a period of time when no timer is lapsed, arbitration may proceedin a round robin manner or in a manner as discussed below with referenceto FIGS. 9 and 10. For simplicity, FIGS. 9 and 10 describe arbitrationfor queues in an ingress buffer. Alternate implementations of router 102provide such arbitration for queues in the egress buffer. A non-blockingrouter may omit operations (e.g., 908, 914, 1006, 1014) related tostalling a queue in either or both of the ingress and egress buffers.

When grants for an output queue are received (902), the quantity ofgrants may be added (904) to a grant pool associated with the queue. Thetotal quantity of grants (corresponding to a total quantity of space forframes at the receiving end) may be determined (906) as a so calledgrant pool depth. If the queue is associated with a flow that has beenstalled, the frame may be left (910) in the queue (e.g., in the ingressbuffer) and processing continues with another frame (922, 702). If theflow is not stalled, it is determined whether there are sufficientgrants for transmitting a frame from the queue. If not, the flow isstalled (914) by setting a flag (e.g., the flag that is tested at 908).Otherwise, the frame is transferred (916) to the fabric 213 and removedfrom the queue; the grant pool is decremented (918); a transferredquantity counter (TQC) is adjusted (920) and processing continues withanother frame (922, 702).

A method for arbitrating among output queues of the same priority,according to various aspects of the present invention, includes anymethod that enables all other queues of a group of queues to empty asmuch as previously emptied from a queue of the group. For example,method 920 of FIG. 10, on removal of a frame from a first queue (e.g., aqueue associated with a source port) of a group of queues, includesadding the size of the transferred frame to the TQC associated with thecorresponding source. If the TQC for this source has a value not greaterthan zero (1004), no further action is taken (1018, 922) and processingcontinues with the next frame (702). Otherwise, the subflow for thissource is stalled (1006) by setting a flag; the positive extent of theTQC (the difference between the TQC value and zero) is assigned (1008)to a variable called the overrun; and the TQC is set (1010) to zero. Foreach other queue in the group of queues (1012) (assuming all queues inthe group have the same priority for transferring frames to the fabric),the queue status is reset (1014) from stalled (if it was stalled) tonot-stalled; and the overrun is subtracted (1016) from the TQC for thatqueue. When all queues of the group have been considered (loop 1012),processing continues (1018, 922) with the next frame (702).

In an embodiment of system 100 having particular synergies forapplication service providers, storage service providers, and storagearea management, network 101 supports protocols of the type known asSCSI protocols over Fibre Channel protocols. Embodiments of this typeare implemented in accordance with the SCSI-3 family of standards andcompatible specifications described, inter alia, inhttp://www.t10.org/scsi-3.htm and available through NCITS Online Storemanaged by Techstreet 1327 Jones Drive Ann Arbor, Mich. 48105(http://www.techstreet.com/ncits.html), particularly those standardsidentified as “Information technology—SCSI-2 Common access methodtransport and SCSI interface module” (CAM), “Information technology—SCSIArchitecture Model-2” (SAM-2), (SBC), “Information Technology—SCSI BlockCommands-2” (SBC-2), “Information Technology—SCSI Reduced blockcommands” (RBC), “Information Technology—SCSI-3 Stream commands” (SSC),“Information Technology—SCSI Stream commands-2” (SSC-2), “InformationTechnology—SCSI-3 Medium changer commands” (SMC), “InformationTechnology—SCSI-3 Medium changer commands-2” (SMC-2), “InformationTechnology—SCSI-3 Multi-media commands” (MMC), “InformationTechnology—SCSI-3 Multi-media commands-2” (MMC-2), “InformationTechnology—SCSI-3 Multi-media commands-3” (MMC-3), “InformationTechnology—SCSI-3 Reduced Multi-media commands” (RMC), “InformationTechnology—SCSI-3 Controller commands” (SCC), “InformationTechnology—SCSI Controller commands-2” (SCC-2), “InformationTechnology—SCSI-3 Enclosure commands” (SES), “InformationTechnology—Object-Based storage devices” (OSD), “Informationtechnology—SCSI Primary Commands-3” (SPC-3), “FIBRE CHANNEL SwitchFabric-2” (FC-SW-2), “Fibre Channel” (FC), “Fibre Channel Protocol”(FCP), “Information Technology—Fibre Channel Protocol for SCSI, SecondVersion” (FCP-2), and “FIBRE CHANNEL Framing and Signaling” (FC-FS). Inother embodiments, SCSI protocols over protocols other than FibreChannel protocols may be used with ports as discussed above. In otherwords, a router may support virtual SCSI transactions, for example, overa port that supports a protocol such as SCSI Parallel Interface, SerialBus Protocol, IEEE 1384 (Fire wire), SSA SCSI-3 Protocol, ScheduledTransfer, and Virtual Interface all of which are the subject of currentpublic standards and draft standards.

According to the terminology defined in protocols for SCSI over FibreChannel, communication is organized to permit an application client toinvoke tasks to be performed by a device server. The communication modelgenerally includes a request from the application client to the deviceserver and a response from the device server back to the applicationclient. A request may be either for device service or for link service.Each task may be part of a task list maintained by the device server. Atask may be invoked, specified, and controlled by a series of commands(e.g., linked commands) communicated by the application client to thedevice server. According to this model, a member may have multipleapplication clients and each application client may have multipleinitiators. Communication from the application client is generallydirected to a target that may have multiple device servers and eachdevice server may act as a responder.

As discussed above, communication comprises transactions comprisingframes. As defined under SCSI protocols, the communication (e.g.,including commands, data, status, and acknowledgements) comprises SCSII/O operations. As defined under Fibre Channel protocol (FCP), each SCSII/O operation is accomplished by a Fibre Channel exchange. Whereas anI/O operation includes a request and a response, an exchange includes aseries of sequences, and each sequence typically comprises severalinformation units. Each information unit corresponds to a frame asdiscussed above. Each sequence of an exchange is transmitted from anoriginator to a responder. If the roles of originator and responder areto be reversed, the originator sends an indication called sequenceinitiative to the responder and the next information unit is expectedfrom the former responder (now an originator).

When a member port is recognized by another port to which the member isconnected, either the port or the member may initiate a login process.Port login is accomplished with FCP IUs with the result that anidentifier for the port of the member is established and associated withthe port of the fabric (e.g., for system 100, port 160 of member 110 isidentified and associated with port 130 of router 102). Port login mayalso result in a quality of service policy being established for thelink between the member and the port (e.g., link 150) and may define ofaffect policies for all paths that include that link. Functions of FCPthat may be included in such a quality of service policy include classof service, intermix mode, stacked connect requests, sequentialdelivery, dedicated service (e.g., connection-oriented), simplex,duplex, camp on, buffered service, priority, preference, initialresponder process associator; capabilities for acknowledgement, datacompression, data encryption, clock synchronization; X_ID interlock,error policy support, categories per sequence, open sequences perexchange, and end-to-end credits (or grants as discussed below).

The correspondence of a typical series of SCSI I/O operations to FCP IUsis described in the aforementioned specifications and is partiallysummarized in Table 6. The target (e.g., a resource as discussed above)may be a block oriented data storage device or a process. Generally, atarget may include many logical units, each logical unit having alogical unit number (LUN). Storage is addressable by a logical blockaddress for a read exchange or a write exchange. A task is an object(e.g., a process) in a logical unit that accomplishes work specified bythe command or by a sequence of commands.

TABLE 6 SCSI I/O Operation Primitive FCP Exchange Primitive Commandrequest. A Unsolicited command information unit (IU) (e.g., FibreChannel command is specified by a Protocol Command: FCP_CMND). AnFCP_CMND IU command descriptor block includes a CDB and may include acommand reference number (CDB) in an initial frame of a (CRN) to assuresequential performance of commands by a task. request. Data deliveryrequest. Data descriptor IU (e.g., FCP Transfer Ready: FCP_XFER_RDY).Used in a write exchange to inform the initiator that the responder isready with a buffer to receive a particular block from the initiator.Data delivery action. Solicited data IU (e.g., FCP_DATA). Used totransfer data in a read or write exchange with a storage device. Fordata exchange with a process, the send and receive commands are definedanalogously. Send Command Complete. Command status IU (e.g., FCPresponse: FCP_RSP). Used to indicate that a SCSI command has beencompleted. Request or Acknowledge Confirmation IU (e.g., FCP_CONF).command completion.

SCSI commands include, inter alia, inquiry, report LUNs, block commands(e.g., read, write, send, and receive) and extended copy. An inquirycommand provides the initiator with parameters of the target or acomponent logical unit of the target, such as device type forcompatibility to receive various SCSI commands. Parameters may includeend-to-end credits (or grants) allocated by the target to the initiatorfor a particular logical unit, process, and/or task. A request to reportLUNs provides the initiator with a list of logical unit numbers of aspecified target. An extended copy command directs data from one set oflogical units to be copied to another set of logical units (or to thesame set of logical units).

SCSI is considered an upper level protocol (ULP) and Fibre Channel alower level protocol (LLP). The lower level protocols include: thephysical interface including media, transmitters, receivers, and theirinterfaces (FC-0); the transmission protocols including serial encodingand error control (FC-1); the transport protocols including frameformat, sequence definitions, transfer of data blocks, and classes ofservice (FC-2); and services concerning several ports at a node (e.g.,operations on a hunt group) (FC-3). The upper level protocols (FC-4)generally include application protocols such as SCSI.

An information unit is transported as a frame. A frame is defined as anFC-2 construct that includes signals recognized as: a start of frame(SOF), a payload, and an end of frame (EOF). For an information unit,the payload is further defined to include an FC-2 header, an FC-2payload, and a cyclic redundancy check (CRC). Further, for aninformation unit, the FC-2 payload includes one or more optionalheaders, an FC-4 header and an FC-4 payload. The information conveyed bythe various portions of an information unit is described in Tables 7 and8, below. Each frame is formed so that the beginning and extent of eachof these portions is determinable under the conventions of theprotocols. Generally, parsing refers to determining the beginning,extent, and meaning of portions of a frame; and formatting generallyrefers to arranging data for transmission as a frame by placing data inthe order defined by the protocols.

A flow, as discussed above may correspond to an exchange identifier(X_ID) comprising an S_ID and a D_ID. A fully qualified exchangeidentifier (FQXID) further includes an initiator identifier, a targetidentifier, an OX_ID, and an RX_ID. The FQXID (as defined in the FibreChannel specifications) is not a complete I_T_L nexus (as defined in theSCSI specifications) comprising an initiator identifier, a targetidentifier, and a logical unit identifier; or, an I_T_L_Q nexus,comprising an initiator identifier, a target identifier, a logical unitidentifier, and a task identifier or tag. A subflow, as discussed abovemay correspond to an I_T_L nexus or an I_T_L_Q nexus.

TABLE 7 FC-2 FC-4 SCSI Delimiter Start of SOF frame Payload FCP R_CTLHeader F_CTL CS_CTL PRIORITY DF_CTL TYPE OX_ID RX_ID SEQ_ID SEQ_CNT S_IDD_ID RO FCP Network Payload header Association header Device header FC-4header LUN CRN Task attributes Task management R-W-Add CDB OP_CODE LBAXFER_L PARAM_LIST_L ALLOC_L CONTROL FCP_DL Data Error CRC Control CodeDelimiter End of EOF frame

TABLE 8 Field Description SOF Any of several ordered sets that indicatethe beginning of a frame. Each start of frame may identify a type offrame to facilitate parsing (e.g., first frame of a sequence, other thanthe first frame of a sequence, class of service, or type of sequencethat follows based on class of service). R_CTL Routing controls.Includes information category describing the information unit assolicited or unsolicited as control, data, command, data- descriptor, orcommand-status. May identify the frame in cooperation with TYPE as linkcontrol (e.g., ACK), extended link services, or a data frame.Information category may identify frames as FCP_CMND, FCP_XFER_RDY,FCP_DATA, FCP_RSP, and FCP_CONF. F_CTL Fabric controls. May specify thatthe frame is from an initiator vs. a recipient; from an originator vs. aresponder; whether the frame is part of a sequence that is the first,last, or neither the first nor the last sequence of the exchange; andwhether the frame is the last vs. not the last frame of a sequence.Fabric controls may further specify if a transfer of sequence initiativeis to take place. Fabric controls may include a flag that specifieswhether to include PRIORITY in place of CS_CTL. CS_CTL Class specificcontrols. For example, Class 1 is for a connection-oriented servicebetween initiator and target; Class 2 is for a connectionlessmultiplexed service with acknowledgement; Class 3 is for aconnectionless multiplexed service without acknowledgement (e.g. withpossible preference indication); Class 4 is for a virtual circuit thatprovides fractional bandwidth between communicating ports, in-orderdelivery, and acknowledgment; and Class 6 is for multiple simultaneousconnection- oriented services between the same two ports. Class 1controls may indicate simplex or duplex. Class 1 and class 6 controlsmay indicate stacked connect request, camp on, and/or bufferedfunctions. Class 2 and class 3 controls may indicate priority delivery(e.g., a 1-bit value for preference on/off). Class 4 controls mayspecify a virtual circuit identifier VC_ID. A QoSF associates S_ID,D_ID, and VC_ID to identify all frames to which the guaranteed bandwidth(and latency) apply. PRIORITY An integer value (e.g., seven bits)indicating to a router having more than one queue and a serving processthat must choose from several queues (e.g., input port queues,processing queues, output port queues) which of several queues toservice next. The PRIORITY value may include a PREEMPTION bit forrudimentary high/normal or normal/low priority determinations. DF_CTLData frame controls. May specify whether or not the FCP payload includesoptional headers. TYPE Data structure type. May indicate communicationprotocol, for example, SCSI, SNMP, IP, internal FC-SW, or VI. Mayindicate IU types for that protocol. For example, for a SCSI command,TYPE in cooperation with R_CTL indicates the frame is formatted toconvey any SCSI command (e.g., FCP_CMND having a CDB), to convey data,or to convey an extended link service request (e.g., FLOGI, PLOGI, orRTIN). OX_ID Originator's exchange identifier. May be assigned by anFC-4 process (e.g., a ULP). RX_ID Responder's exchange identifier. Maybe assigned by an FC-4 process (e.g., a ULP). SEQ_ID Sequenceidentifier. May be assigned by an FC-4 process (e.g., a ULP). SEQ_CNTSequence count indicates a serial number of the frames having the sameSEQ_ID. Useful for maintaining frames in-order. S_ID Source identifier.Identifies the network port that transmitted the frame. Typically a24-bit number that identifies the initiator. It may be divided intothree 8-bit portions designating domain (an identifier of a router,e.g., router 102), area (an identifier of a physical output port of therouter, e.g., 130), and loop address (an identifier of a resource on aloop serviced by the port). D_ID Destination identifier. Identifies thenetwork port intended to eventually receive the frame. Typically a24-bit number that identifies the target. It may be divided into three8-bit portions designating domain, area, and loop address. The D_ID mayspecify a group address or a well known address. Well known addressesare reserved values for, inter alia, a multicast server, a clocksynchronization server, a security key distribution server, a timeserver, a directory server, a broadcast alias, an alia server, amanagement server, a quality of service facilitator (QoSF), a fabriccontroller (e.g., managing process 204), or a fabric port. RO Relativeoffset. A displacement in bytes describing the first byte of a payloadrelative to a data buffer that was read to form the payload or a databuffer that will be written when the payload is delivered to itsdestination. The relative offset may be designated as random orcontinuously increasing for different information categories. Networkheader Includes, respectively for S_ID and for D_ID of the FCP header, adesignation of an authority that assigned a name (e.g., CCITT, IEEE) anda name identifier (e.g., 60-bit value, WWPN). Association headerIncludes, respectively for S_ID and for D_ID of the FCP header, aprocess identifier (e.g., a 56-bit object reference used with CORBA).Device header Provides to a ULP additional identification of theexchange already identified by the FCP header. LUN Logical unit number.Identifies a resource of the member at the destination network port. Maybe a WWPN or a suitable 64-bit identifier. CRN Command reference number.May be used to assure that SCSI commands are performed in-order. Taskattributes May specify which task queue, type of task queue, and theposition in that task queue at which the task defined by this exchangeis to be inserted. For example, simple queue, head of queue, orderedqueue, ACA queue, and untagged task. Task management Specifiesoperations on a logical unit and/or a task queue associated with alogical unit, such as: abort task set, clear task set, reset a logicalunit, reset a target, and clear an ACA. R-W-Add May indicate by a singlebit (facilitating parsing) whether the CDB is for a read command or awrite command or neither (and analogously a send or receive commandreferring to a target process). May also specify an additional lengthfor an extended length CDB. An extended length CDB may convey a virtualLUN through a fabric. CDB Command descriptor block. OP_CODE Operationcode. Specifies the SCSI command (e.g., PLOGI, REPORT LUNS, READ, WRITE,SEND, RECEIVE) LBA Logical block address. May include page number (e.g.,11 bits), sector number (e.g., 11 bits), and block offset (e.g., 11bits) designating a 512- byte block that is a portion of a sector.XFER_L Transfer length. PARAM_LIST_L Parameter list length. ALLOC_LAllocation length. CONTROL May specify whether the command is part of aset of linked commands. May also indicate controls for a cachemaintained by the device server, for example, specifying to disable pageoutput from the cache (DPO), and force unit access (FUA) to supercedecache access. CRC Any code, typically a cyclic redundancy check code,that may be used by the receiver to verify the integrity of all or aportion of the transmitted payload. EOF Any of several ordered sets thatindicate the end of a frame. Each end of frame may identify a type offrame to facilitate parsing or link control functions (e.g., terminationof a class 4 circuit, content of the frame is invalid, last frame of asequence, or other than the last frame of a sequence).

Table 9 briefly describes some of the contents of FCP IUs thataccomplish a SCSI write command. The IUs in Table 9 form one exchange.Each IU is a sequence of that exchange. For each IU, the S_ID identifiesthe transmitting port (the originator, generally having sequenceinitiative) and the D_ID identifies the receiving port. These alternate,though the identity of the initiator and the target are unchangedthroughout the exchange. Note that the LUN is conveyed in the FCP_CMNDCDB and is not included in the FCP_XFER_RDY, FCP_DATA, or FCP_RSP IUs.To implement a quality of service at the logical unit level, the logicalunit number corresponding to the exchange must be recorded from theFCP_CMND IU; and, referred to for other IUs of the exchange.

TABLE 9 Information Unit Brief Description of Selected ParticularContents FCP_CMND SOFi2 (Class 2); OX_ID; S_ID is originator of thisexchange; S_ID is initiator of this FCP_CMND sequence; This is the firstframe of the sequence; End sequence (i.e., this frame is the end of theFCP_CMND sequence); Transfer sequence initiative to responder; EOFn(normal); ACK SOFi2; RX_ID; EOFt (terminate); FCP_XFER_RDY SOFi2; FQXID;S_ID is responder in this exchange; S_ID is initiator of thisFCP_XFER_RDY sequence; End sequence; Transfer sequence initiative toinitiator; RO from LBA to be written; EOFn; ACK SOFi2; EOFt; FCP_DATA(one of SOFi2; FQXID; Originator; Initiator; End sequence; Transfersequence several, each initiative; data to be written at RO from LBA;EOFn; followed by an ACK) ACK SOFi2; EOFt; FCP_RSP SOFi2; FQXID;Responder; Initiator; Last sequence of this exchange; End sequence;Transfer sequence initiative; EOFn; ACK SOFi2; EOFt;

The terminology used to describe system 100 may differ somewhat from theterminology defined in the FCP specifications. In the FCPspecifications, a fabric is an entity having ports that routes framesbetween its ports using only the D_ID from the FC-2 header. A path is aroute through the fabric from a source to a destination. A path mayinclude one or more hops. A fabric may include multiple switches, eachswitch being an entity defined as a fabric element having ports, a pathselector, an address manager, a fabric controller, a router, and aswitch construct that transports frames between ports as directed by therouter. A router, as defined in the FCP specifications, is an entitywithin a switch that determines for each received frame what port todirect the received frame so as to accomplish a connectionless delivery.System 100 is described herein in broad terminology as an example of animplementation according to various aspects of the present invention. Toprepare an FCP SCSI implementation according to various aspects of thepresent invention, the specific functions of the FCP and SCSI protocolspecifications are generally mapped as an instance of the functions andstructures described herein that may bear the same or differentnomenclature. Access controls discussed with reference to system 100 areenforced by a router or a proxy, whereas access controls under SCSI andFCP protocols may be enforced by the target (e.g., a device server).

As discussed above, routing information as determined by anadministrating process or a managing process may include an I_T_L nexus(or I_T_L_Q nexus) for a virtual or nonvirtual member or resource. Forexample, a managing process may launch a proxy for each I_T_L or I_T_L_Qnexus that refers to a virtual identifier (e.g., a virtual member, or avirtual LUN of a nonvirtual or virtual member).

A router, according to various aspects of the present invention,includes any switch that implements architecture 200 as discussed above.In one implementation, a router includes a supervising processor and aplurality of routing processors, the routing processors being coupled toa fabric comprising a ring network. In another implementation, thefunctions of routing process 208 are implemented in an integratedcircuit comprising a frame processor, multiple interfaces for ports tonetwork 101, and circuits that implement a serial slice of the ringnetwork of the fabric. For example, router 102 of FIGS. 11-14 includesmanaging processor 1112; local console 1102 coupled to managingprocessor 1112; remote console 1106 coupled via bus 1104 to managingprocessor 1112; host bus adapter 1140 coupled 1142 between managingprocessor 1112 and a frame I/O port 1198; erasable programmable memory(EPM) 1114 coupled to managing processor 1112; random access memory 1116coupled to managing processor 1112; and a plurality of routing circuits1150-1152 coupled to managing processor 1112 via local area network(LAN) bus 1132, EPM bus 1134, and test bus 1136. A ring 1170 connectsthe plurality of routing circuits to provide functions of fabric 213.Each routing circuit 1150-1152 includes supervising processor 1160,memory circuit 1162, and a plurality of port logic circuits 1186-1188.Each port logic circuit provides several frame I/O ports 1192 and 1194(for routing circuit 1150); and frame I/O ports 1196 and 1198 (forrouting circuit 1152). In one implementation, a router 102 having 20frame I/O ports is formed on one printed circuit board (excludingconsoles 1102 and 1106 and network 1104).

A managing processor includes any stored program computer circuit thatmanages operations of one or more supervising processors by acceptingpaths from an administrating process, providing reports to anadministrating process, providing routing information to one or moresupervising processes, governing operation of one or more supervisingprocesses to assure policy effectivity on one or more links, serving asa proxy, and operating a cache—all, for example, as discussed above. Forexample, managing processor 1112 may include any computer circuit havinginterfaces to memory and communication buses and cooperating with a hostbus adapter. Managing processor 1112 provides a conventional interfaceto memory for program storage and work space. Program memory, EPM 1114,may include any persistent store (e.g., erasable programmable memory,disk, and RAM) for storage of instructions for processes described withreference to FIG. 4, an operating system, and suitable device drivers.Workspace memory, RAM 1116, may include any memory circuit (e.g., RAM,EPM, cache memory, or disk) for storage of data described with referenceto FIG. 4. Managing processor 1112 supports one or more consoles 1102and 1106 that accept input from an operator. Managing processor 1112communicates with host bus adapter 1140 via line 1144 to send andreceive frames. Managing processor 1112 communicates with supervisingprocessors via a bus (e.g., a local area network) such as LAN 1118,1132, 1152. Managing processor 1112 transfers data for image updatesfrom EPM 1114 to routing circuit memory (e.g., 1162) via EPM bus 1120,1134, 1154. Managing processor 1112 communicates with supervisingprocessors for diagnostic, test, and watch dog purposes via test bus1122, 1136, 1156. In one implementation, LAN 1132 has physical andlogical capabilities of the type known as Ethernet (see IEEE Std.802.3), EPM bus 1134 has the physical and logical capabilities of thetype known as a PCI bus (see PCI Local Bus Specification by PCI InterestGroup, Portland Oreg.), and TEST bus 1136 has the physical and logicalof a conventional asynchronous serial communication interface (e.g.,using ASCII character codes for commands, addresses, status, and data).Managing processor 1112 may control fans, power supplies, EPM and otherdevices using a two wire serial interface of the type known as an I²C(see I²C Bus Communication by Philips Semiconductor).

In one implementation, managing processor includes an Intel Socket 370440-BX chip set hosting an open sources operating system, for example,of the type known as Linux.

A console provides a GUI for an operator (human or automated) to specifyparticular values for router configuration and for displaying status,reports, error messages, warnings, and prompts. For example, localconsole 1102 is coupled in any suitable manner to managing processor1112. At any time one or more remote consoles 1106 may be coupled vianetwork 1104 to managing processor 1112. Local console and remoteconsoles are functionally similar in displays and controls. For example,these consoles may be implemented with any client computer (e.g., aterminal, workstation, or personal computer).

A host bus adapter provides an interface for frame communications (e.g.,as described above with reference to SCSI). For example, host busadapter 1140 includes an interface to connect to a physical port ofrouting circuit 1150 via line 1142. Host bus adapter 1140 may transferframes or portions of frames after parsing and error correction tomanaging processor 1112 or RAM 1116 (e.g., directly via lines notshown). Host bus adapter 1140 may transfer data for frames or portionsof frames (e.g., payloads) from managing processor 1112, EPM 1114, orRAM 1116 and perform frame assembly in any suitable manner (e.g.,determining header and error control data for one or more frames). Datatransfer may utilize direct memory access techniques and/or descriptorsas discussed below. In one implementation, the managing processor andhost bus adapter are provided on a single integrated circuit substratethat provides one or more multi-conductor parallel digital interfacesfor coupling to consoles, memory, and routing circuits.

A routing circuit includes any circuit that routes frames according toidentifiers (e.g., addresses) as discussed above. For example, router102 may include one or more routing circuits 1150-1152 each coupled toat least one managing processor 1112 for performing supervising androuting processes as discussed with reference to FIGS. 5 and 6.

Fabric 213 of router 102 (FIG. 2) is implemented by ring 1170 shownfunctionally as one line though any suitable number of bus orpoint-to-point conductors are used in various implementations. In oneimplementation a router 102 has only one routing circuit 1150,simplifying design of ring 1170. Ring output 1172 from one port logiccircuit 1186 is coupled (directly or through other port logic circuits)to ring input 1174 of a subsequent port logic circuit 1188.

Alternatively, ring 1170 extends between routing circuits so that eachrouting circuit 1150 communicates with each other routing circuit 1152.A ring output of a port logic circuit 1188 of a routing circuit 1150 iscoupled to ring input of a subsequent port logic circuit of a subsequentrouting circuit 1152. A ring permits frame I/O from any physical port ofrouter 102 (e.g., ports 1192, 1194, 1196, and 1198) to be routed to orfrom any other physical port of router 102.

Each routing circuit supports a multiplicity of router ports, generallyof identical functionality. Each router port may be coupled for frameI/O to any one of a member of system 100, another router of system 100(e.g., an expansion port), a console as discussed above, or a host busadapter 1140.

A supervising processor includes any stored program computer circuitthat manages operations of one or more port logic circuits by acceptingmaps from a managing process, providing status to a managing process,providing and updating routing information to one or more routingprocesses, acting on link service requests, providing link servicereplies, advising proxy processes (e.g., of link service actions, linkstate, network traffic, events, or configuration) that may affectoperations performed by the proxy, managing shared use of communicationand memory facilities shared by routing processes, and governingoperation of one or more routing processes to assure policy effectivityon one or more links—all, for example, as discussed above. For example,supervising processor 1160 may include any computer circuit havinginterfaces to memory and communication buses. Supervising processor 1160may provide any conventional interface to port logic circuits andmemory. For example supervisory (SUPRV) bus 1176, 1164 couplessupervising processor 1160 to any number of port logic circuits1186-1188 and to memory circuit 1162. In one implementation, SUPRV bus1164 has physical and logical capabilities of the type known as PCI busas discussed above. Supervising processor 1160, any port logic circuit1186-1188, or memory circuit 1162 may become master of SUPRV bus 1176for directing data transfer operations. By permitting bus masterfunctions from any port logic circuit, efficient use of SUPRV bus 1164results. Such use may assure policy effectivity for a particular port.Port logic circuits 1186, 1188 and memory circuit 1162 may include CSRs(e.g., for DMA control configuration) that are mapped to addresses ofthe PCI bus.

In one implementation, supervising processor 1160 includes a single chipcomputer having an Intel x86 compatible processor, PCI, I²C, flashmemory, GPIO, and memory bus interfaces of the type marketed by AMD asmodel SC520. Supervising processor 1160 may perform a real timeoperating system of the type known as Linux as discussed above.

Preferably, operating systems in the managing processor and supervisingprocessor support interprocess communication between several of theseprocessors. For example, in one implementation, interprocesscommunication is implemented using Common Object Request Broker Agent(CORBA) software of the type that allows processes and subprocesses tobe identified (e.g., by an object reference). An administratingprocessor may obtain the services of any object made available via CORBAhosted on any managing processor or supervising processor. For example,managing process 204 may include objects for access control listmaintenance, policy value maintenance, group membership maintenance,zone membership maintenance; and, supervising process may includeobjects for statistics probes (e.g., permitting control of statisticsgathering as to what to gather and when), and routing table maintenance.Managing and supervising processes may include agents that define APIsfor one or more objects to simplify inter-object communication andcontrol.

A memory circuit provides multiple access to routing information, statusinformation, and configuration information. For example, memory circuit1162 is coupled to supervising processor 1160 via SUPRV bus 1177, 1164(e.g., for obtaining routing information as updates of images) and toport logic circuits 1186-1188 via ROUTE bus 1178, 1166 (e.g., forresponding to demands from port logic circuits for routing information,status of other port logic circuits, and configuration information).Each port logic circuit may be coupled to memory circuit 1162 via anindependent channel effected via dedicated lines (e.g., separate buses)or dedicated time slots on a multiplexed bus.

A port logic circuit includes any circuit that provides at least aphysical interface to one or more frame I/O ports, cooperates with otherport logic circuits 1186-1188 via a fabric, and accesses memory forrouting information. In one implementation, a port logic circuitprovides a logical interface for each frame I/O port so that asupervising process or a routing process may send and receive data via alogical port using an API in some ways independent of frame structureand signaling protocol of the physical port. In one implementation, eachport supports both Ethernet and Fibre Channel frame structures andsignaling protocols so that the same routing process and the samesupervising processes may communicate with ports regardless of whetherthe port is from time to time physically connected to an Ethernet linkor a Fibre Channel link. For example, a group of frame I/O ports 1192supported by port logic circuit 1186 may include a physical interfacefor each of four links, each link being compatible with Ethernet orFibre Channel.

A supervising processor in one implementation according to variousaspects of the present invention includes a first bus for a processor,memory, and interfaces; a second bus; and a bridge between the first busand a second bus. The processor may have exclusive control of the firstbus to simplify program operations performed by processes hosted by theprocessor. The processor may cooperate with other processorsintermittently controlling and relinquishing control of the second busto facilitate maximum efficient use of the capacity of the second bus.For example, supervising processor 1160 includes bus 1204 (e.g., asuitable multi-conductor parallel digital bus) coupled to processor1202, program store 1206, data memory 1208, serial controller 1210,persistent store 1212, I/O bus controller 1214 coupled between bus 1204and bus 1216 (e.g., a PCI bus) to perform functions of a bridge; and LANcontroller 1218 coupled to bus 1216. Supervising processor 1160 iscoupled to TEST bus 1136 via line 1156, EPM bus 1134 via line 1154, LANbus 1132 (e.g., Ethernet) via line 1152, and SUPRV bus 1164 via line1176. Any conventional circuits may be used to implement the functionsof supervising processor 1160 including any mix of memory: volatile andnonvolatile (e.g., erasable programmable memory). Nonvolatile memory maybe used to store programs (e.g., EPM of store 1206) and/or configurationvalues (e.g., persistent store 1212). Configuration values may includeany suitable values that facilitate the assembly of routing circuit 1150in commercially desirable configurations utilizing similar components(e.g., populating a printed circuit assembly to various extents for avariety of router models). For example, configuration values may includethe number of port logic circuits, the electrical position of adistributing circuit in the fabric ring, the addresses (via SUPRV bus1164) of installed port logic circuits, addresses that describeallocations of memory 1162 to port logic circuits (e.g., forconfiguration of port logic circuits and communication betweensupervising processor 1160 and particular port logic circuits), anddefault port characteristics (e.g., physical interface capabilities orphysical port identifiers).

A memory circuit in one implementation according to various aspects ofthe present invention provides routing information (e.g., includingcross reference information) by the cooperation of shared random accessmemory, content addressable memory, and random access memory that isaddressed at least in part by data recalled from content addressablememory. For example, memory circuit 1162 includes memory controller1302, multi-purpose memory 1304 coupled via line 1303 to memorycontroller 1302, content addressable memory (CAM) 1306 coupled via lines1305 and 1307 to memory controller 1302 and coupled via line 1309 torandom access memory (RAM) 1312. Random access memory 1312 is alsocoupled to memory controller 1302 via line 1313.

Memory controller 1302 is coupled to SUPRV bus 1164 via line 1177 andcoupled to ROUTE bus 1166 via line 1178. Generally, configuration valuesreceived via SUPRV bus 1177 are stored in multi-purpose memory 1304 vialine 1303. Routing information (e.g., maps, image data, and updates)received via SUPRV bus 1177 is stored in CAM 1306 and RAM 1312. When arequest for routing information is received via ROUTE bus 1178 by memorycontroller 1302, memory controller 1302 presents a query (e.g., a tag)via line 1305 to CAM 1306. Tags presented to the CAM may have one of 8types as indicated by a 3-bit field. Typical queries are described inTable 10.

TABLE 10 Purpose of Query Query Components Flow lookup tag type; sourceidentifier (e.g., value from S_ID field of received frame); destinationidentifier (e.g., value from D_ID field of received frame—may bevirtual); class of service (e.g., indicated by SOF analyzed by 1406,value of CS_CTL field of received frame); protocol identifier (e.g., asdetermined by parser 1408 mask/pattern comparisons); input physical portidentifier (determined by 1406); input physical port type (e.g., portspeed, signaling protocol; determined by 1406); configuration settings(e.g., CSR values set by processor 1424); Subflow tag type; all fieldsof a flow lookup; flow identifier (from flow lookup CAM lookupassociated data); Virtual flow tag type; all fields of a flow lookup anda subflow lookup; virtual member lookup identifier (e.g., a destinationport identifier such as a D_ID field value from the received frameformatted to be recognized as a virtual port identifier); a virtualresource identifier (e.g., a LUN field value from the received framerecognized as virtual by association with the virtual memberidentifier);

CAM 1306 responds to a query by providing a flag on line 1307 indicatinga successful search. When the search is successful, data on line 1309provides an address to RAM 1312. RAM 1312 responds to the address byproviding additional query results as data on line 1313 described inTable 11. When the search is successful, data from RAM 1312 (also calledCAM associated data) on line 1313 is valid. CAM associated data isdescribed in Table 11.

TABLE 11 Query RAM 1312 Response (line 1313) Flow lookup priority (e.g.,traffic class); output physical port identifier; flow identifier (e.g.,assigned by routing processor upon receipt of FCP_CMND from initiator, ahashed version of source and destination world wide port names createdby the parser); flag for subflow lookup required; output port speed;action code (e.g., 2 bits); flag for stall; flag for default route;marking for output frame (e.g., revised CS_CTL value); mid-switch stage(e.g., 4-bit hop count); statistics sample interval; statistics index(e.g., identifies which counter should be used for countable eventsassociated with this flow); Subflow resource identifier (e.g., forSCSI-3 protocol on FCP, logical unit number lookup (LUN); for VirtualInterface, VI handle of participating process); flag for routingprocessor action associated with the type of frame as determined byparsing; process identifier for routing processor (e.g., a jump vector,or object reference); Virtual flow page table start address; page size(allows programmable page sizes); sector lookup size (allowsprogrammable sector sizes); shift value (used to determine page boundarycrossing); flag set to indicate frames to this virtual LUN should bediscarded (e.g., LUN not defined, or does not presently exist); flag setto indicate the virtual LUN is busy (e.g., frame should be stalled orrouted to managing processor for routing by proxy); flag set to indicaterouting for the virtual LUN is disabled (e.g., frame should be routed tomanaging processor for routing by proxy);

A method for revising the configuration of a plurality of routingprocessors includes in any order: (a) for each virtual entity and eachrouting processor to be reconfigured (e.g., each processor that usesrouting information implementing routing for a particular virtualentity), setting a flag to indicate that routing for the virtual entityis disabled; (b) routing to a managing processor for dispositionsubsequently received frames (e.g., both control frames and data frames)that indicate the virtual entity as a destination; (c) enabling proxyprocesses performed by the managing processor to respond to or routesuch subsequent frames; (d) storing new routing information for avirtual entity in a memory accessible by a routing processor; and (e)clearing the flag(s) in router(s) previously set so as to enable routingof traffic for the virtual entity in accordance with the new routinginformation. New routing information may be stored in one router (e.g.,for access by one or more routing processors or frame processors) or inseveral routers facilitating routing for one or more virtual entities(e.g., for distributing the processing burden of virtualization,security, or redundancy).

Memory controller 1302 provides a response on ROUTE bus 1178 to eachquery received on ROUTE bus 1178. Information conveyed by such aresponse includes the data described in Table 11 without the address ofRAM 1312. When the query is accompanied by an identification of thequery (e.g., a worklist pointer value, or an identifier of a queue fromwhich the submitter composed the query), the response may be accompaniedwith a suitable corresponding identification of the query.

In one implementation, CAM 1306 includes a content addressable memory ofthe type marketed by Lara Networks Inc. as model LNI7020. Memorycontroller 1302 includes a CAM controller of the type marketed by LaraNetworks Inc. as model LNI8010.

Memory controller 1302 may provide address mapping so that a particularrouting process 208 may access a unique portion of multi-purpose memoryfor its particular configuration or communication needs; and allinstructions for separate routing processes may be installed in separateport logic circuits using identical instructions in each routingprocess. By providing address mapping, configuration control of routingprocess instructions is simplified. In other words, multi-purpose memory1304 includes an area reserved for each routing process 208. Eachrouting process (e.g., a port logic circuit may have one or more routingprocesses) may have a reserved area (not necessarily a contiguous rangeof addresses). Contents of multi-purpose memory 1304 are described inTable 12.

TABLE 12 Variable Stored in a Reserved Area of Multi-purpose Memory 1304Description Shared tables Each table shared by several routingprocessors. Tables may include data structures for the following:context table, virtual context table, port table, page table, and sectortable. Configuration values Each configuration table may be reserved foruse by one routing for each routing processor. Values may includeinitial values for CSRs and software for processor use by the routingprocessor. Supervisor queue for Each queue may be reserved for use byone routing processor and a each routing suitable supervising processor.May include entries for link service processor requests, link servicereplies, and frames to be analyzed and/or routed by the supervisor.Reserved tables Tables used by only one routing processor. In variousimplementations any one or more of the context, virtual context, port,page, and sector tables may be segmented to reduce memory utilization orimprove access.

A port logic circuit includes any circuit that performs a routingprocess as described above. Implementation of such a routing circuit mayinclude one or more of a stored program computer circuit, a microcodedstate machine circuit, and a combinatorial logic circuit (e.g., withcounters and/or state variable storage). The stored program computercircuit and/or microcoded state machine circuit may employ EPM tofacilitate implementing routers 102 in various configurations (e.g.,supporting a wide variety of signaling and communication protocols inone router). By performing different portions of the routing process indifferent circuits, a relatively high degree of parallel processing mayresult with concomitant non-blocking frame I/O processing capacity. Inone implementation, a port logic circuit, inter alia, performs flowrouting, performs subflow routing, performs routing of virtual dataframes, parses frames to facilitate routing virtual control frames to aproxy, gathers and reports traffic statistics, assures specified qualityof service by arbitrating among data flows on ingress and/or on egress,facilitates frame routing via a ring, and communicates with asupervising process via a supervisor queue and interrupts—all, forexample, as discussed above. For example, port logic circuit 1186 ofFIG. 14 includes distributing circuit 1402, egress buffer 1414,arbitrating circuit 1405, media interface circuit 1406, parser 1408,ingress buffer 1410, index 1420, submitter 1422, frame processor 1424,access circuits 1404, cross reference circuits 1425, virtual outputqueue controller 1428, dequeue logic 1412, and statistics store 1426.

A distributing circuit includes any circuit that performs the functionsdescribed above with reference to a fabric. For example, distributingcircuit 1402, implements functions of a network implementation of fabric213, in particular a ring network. Distributing circuit 1402 receivessignal RING-I on line 1170, recognizes frames addressed to the physicalports supported by port logic circuit 1186, places at least the payloadof such frames (received via line 1430) in egress buffer 1414, respondsto signal CONTROL on line 1432 to avoid egress buffer overflow, providessignal CONTROL on line 1436 to synchronize and/or enable sending,receives (via signal DATA 1434) at least the payload portion of framesto send to other physical ports, and provides signal RING-O on line 1172for input to a subsequent port logic circuit as discussed above.

Frames conveyed by signals RING-I and RING-O may also include one ormore of timing signals, header information, error detection andcorrection information, signals indicating priority, quality of service,class of service, traffic class and distribution capacity allocation andarbitration controls. Distributing circuit may receive frames fromdequeue logic 1412 fully formatted for the fabric network; or,distribution circuit may perform formatting functions. Distributingcircuit may provide frames to egress buffer 1414 complete with allfabric network formatting; or may remove formatting by parsing the frameand providing only portions of the fabric network frame to egress buffer1414. Distributing circuit includes address comparison logic so as todetermine whether a frame received on line RING-I is within an addressrange (e.g., matches all or a portion of a physical port address) and ifso to provide at least the payload to egress buffer 1414. Distributioncircuit 1402 is nonblocking, sends every frame that it receives fromsignal RING-I except those delivered to egress buffer 1414, and sendsevery frame that it receives from dequeue logic 1412. Frames that aresent are provided once on line RING-O 1172.

An egress buffer provides storage for frame payloads to be delivered toa frame I/O port. An egress buffer may include ring buffers formed fromlinked lists so that frame payloads of varying length may be suitablystored and accessed. An egress buffer may have separate ring buffers foreach of several traffic classes for the same physical output port. Anegress buffer may support several (e.g., four) physical ports (e.g.,with parallel control circuitry and shared memory). The payload insertedinto a ring buffer of an egress buffer already includes suitable indiciaof the destination port and is therefore suitable for delivery from theegress buffer without revision. Such indicia of the destination port maydiffer from the physical port identifier corresponding to the frame I/Oport 1192 of this hop (e.g., destination port 166 may be indicated in aframe delivered out of port 133. See FIG. 1).

For example, egress buffer 1414 (comprising combinatorial logic, memory,and a state machine) includes a ring buffer for each of 4 trafficclasses. All frames received from distributing circuit 1402 that arewithin the address range corresponding to frame I/O port 1102 (e.g.,carrying in a header the exact physical port address of a physical portof frame I/O port 1192) are stored in a ring buffer corresponding to thetraffic class indicated in a header of the frame.

An arbitrating circuit identifies at a suitable time each frame payloadto be sent as a frame and determines which of several competing suppliesof frames should be used as a source of supply for the next opportunityto send a frame. For example, arbitrating circuit 1405 (comprisingcombinatorial logic, memory, and a state machine) identified for aphysical output port frame from one of several queues of egress butter1414. Each queue may correspond to a traffic class or other policyvalues (e.g., queue for each of four traffic classes). The identifiedframe (or frame identification) is passed (1438, 1439) from egressbuffer 1414 to a media interface circuit for the physical port.Arbitrating circuit 1405 may use a round robin scheme among queueshaving a non-empty ring buffer and sufficient grants (e.g., requestedand granted by the receiving end of the link according to FC-2) topermit delivery. Arbitrating circuit 1405 may examine a PREFERENCE orPREEMPTION bit and alter arbitration to service marked frames ahead ofother frames. Arbitrating circuit 1405 may also set and monitor timersand service a queue in response to its respective timer. In an alternateimplementation, arbitrating circuit 1405 performs a method similar tomethod 702 discussed above as amended to pertain to an output physicalport as opposed to the fabric (916) and to an egress buffer as opposedto an ingress buffer (910). Memory may serve to retain a respectivegrant pool depth and TQC value for each queue (e.g., for each trafficclass and for each physical port). Thresholds and other arbitrationconfiguration values are set by processor 1424 via line 1463.

A media interface circuit serves as an interface between a physicalnetwork connection and other portions of a port logic circuit. Onoutput, a media interface circuit assures that signaling rules andframing rules of a desired link protocol are met for each physical port.On input, a media interface circuit derives from a frame received from alink a set of signals and information for parsing that are independentof the signaling and framing rules of the link protocol. For example, alink may conform to one of several protocols. After configuration of themedia interface circuit, a parser coupled to the media interface circuitcan parse its inputs without regard to the particular protocol of thelink. For example, media interface circuit 1406 (comprisingcombinatorial logic, memory, and state machine(s)) determines frames foroutput in response to signals received from arbitrating circuit 1405,and after being configured by frame processor 1424 for a particular linkprotocol, adds timing signals, header information, and error detectionand correction information, and sends at a suitable time, a suitableframe on a frame I/O line 1192. On input, after being configured byframe processor 1424 for a particular link protocol, media interfacecircuit 1406 receives frames from frame I/O line 1192, removes timingsignals, analyzes (and suitably strips) header information, and stripserror detection and correction information after use for determiningwhether retransmission should be requested. The results of stripping andanalysis are provided to parser 1408 via line 1446. Error detection andcorrection and retransmission requests may be logged by counters instatistics store 1426. Media interface circuit 1406 may support severalphysical ports (e.g., four) using parallel circuits, one for eachphysical port. Portions of the functions of interface signal generationand detection may be accomplished for each frame I/O line by additionalcircuitry (not shown) that may be implemented external to an integratedcircuit implementation of port logic circuit 1186. In particular, line1142 that couples host bus adapter 1140 to a port may interface directlyto a port logic circuit integrated circuit without conversion to signallevels suitable for use external to router 102. Media access controller1406 in one implementation accomplishes functions defined by FCPspecifications as levels FC-0, FC-1, and at least the FC-AL portion ofFC-2 (e.g., including state machine(s) for buffer-to-buffer flowcontrol, point-to-point communication, dual speed 1 Gigabit/sec and 2Gigabit/sec, auto negotiation) and analogous functions for IEEE 802.31000BTX Ethernet.

An integrated circuit that implements functions of a port logic circuit1186 as discussed above may include four frame I/O ports per parser; oneparser, one submitter, and one filter per frame processor; two frameprocessors serving one virtual output queue controller, one ingressbuffer and one egress buffer; and four output channels to a distributingcircuit. The egress buffer may have 11 queues per output frame I/O port.The frame processors may have a 3-stage pipeline (e.g., fetch, execute,store) similar in some respects to a RISC processor.

A parser, for each received frame, identifies portions of the receivedframe and alerts a frame processor to begin processes that route theframe. A parser may also prepare a query to be submitted for obtainingflow routing information, subflow routing information, or virtual flowrouting information. For example, parser 1408 (comprising combinatoriallogic) receives information about each received frame via line 1446 frommedia interface circuit 1406, stores frame information in ingress buffer1410 via line 1444, identifies particular portions of the stored frameby storing via line 1448 pointers in index 1420, provides notice (e.g.,an interrupt) to frame processor 1424 via line 1468 to initiateprocessing of the frame, and provides information via line 1470 tosubmitter 1422 for preparation of a query. In one implementation, parser1408 and index 1420 provide frame processor 1424 with access to thefirst 128 bytes of every received frame. Different frame types maylocate similar fields in different places (e.g., length of field LBA maycause different location of field FC_DL). Parser 1408 may cooperate withindex 1420 to provide uniform access to particular frame fields (e.g.,RO), accounting for differences in frame formats. Parser 1408 maydetermine a frame format, for example, with reference to SOF and TYPEand direct offsets between pointer values in index 1420 to accomplishsuitable access. In one implementation, parser 1408 classifies frames ofnetwork traffic by identifying field locations of various frame formatsin successive comparisons. For each comparison,:the received frame ismasked (e.g., to select values of TYPE_ and R_CTL) and the masked valueis compared to a pattern. If the result of comparison is successful,pointers to fields are assigned values in accordance with field locationdata. For example, up to eight comparisons may be attempted by selectingin turn a tuple of mask, pattern, and field location data from memoryparser 1408. The result may provide a coded value (e.g., 3-bit protocolidentifier). This memory may be loaded from configuration data byprocessor 1424 via line 1463. By loading this memory from time to timein accordance with configuration data, each port 1192 of router 102 maybe configured to support one of a variety protocols.

As parser 1408 commits memory (e.g., in index 1420 and ingress buffer1410) parser 1408 provides signals 1446 that direct media access circuit1406 in responding to requests for buffer-to-buffer grants.

An ingress buffer, inter alia, provides storage for information destinedto be assembled into a frame to be sent via the fabric to a frame I/Oport. Such information may have been derived from a frame received froma link; or, may have been determined by a frame processor forcommunication with a managing process or an administrating process. Aningress buffer may also provide storage for information destined to bepassed to (or as received from) a supervising process. Such informationmay have been derived from a frame received from a link (e.g., a linkservice request); or, may have been determined by a supervising processfor communication with a network member (e.g., a link service reply).For example, ingress buffer 1410 (comprising combinatorial logic andmemory) receives data to be stored from parser 1408 via line 1444, datato be stored from access circuits 1404 (e.g., DMA controller 1491) vialine 1443, and data from frame processor 1424 via line 1456. Ingressbuffer 1410 provides data from storage to dequeue logic 1412 via line1440, data from storage to access circuits 1404 (e.g., DMA controller1491) via line 1443, and data from storage to frame processor 1424 vialine 1456. Data in storage may be organized in a ring buffer (e.g.,linked lists) for each output queue. In an implementation havingmultiple traffic classes per output queue, data in storage may beorganized in a ring buffer for each traffic class of each output queue.In a preferred implementation, policy values are effected on each flow,subflow, and virtual flow in part by enqueueing fames in accordance withthe physical port identifier (of this router) that received the frame,the physical port identifier (of this router) to which the frame isdestined to be sent, and one or more policy values (e.g., one at fourtraffic classes).

Ingress buffer 1410 may include circuits for adding frame formattingsuitable for fabric network frames. A fabric network frame may enclosethe header and payload of a frame received from a frame I/O port andthereby provide prepended header. The prepended header may includedestination physical port identifier or address (e.g., to be read by anyegress buffer 1414 in any port logic circuit on ring 1170 (e.g., fabric213)), priority, destination port speed, source physical port identifieror address, and flags designating, for example, whether this is amulticast frame.

An index provides pointers and may provide other descriptors ofsignificant portions of data stored in an ingress buffer. Descriptorsmay include starting addresses, lengths, flags, and values indicatingthe type of processing prescribed by a parser or supervising processor.For example, index 1420 (comprising combinatorial logic and memory)receives values for storage in its memory from parser 1408 via line1448. Frame processor 1424 reads via line 1462 index 1420 to addressingress buffer 1410 via line 1458 and thereby access any desired framedata. Frame data descriptors stored in index 1420 may be read byprocessor 1464 via line 1462. In one implementation, index 1420 includesmemory organized in rows (or slots). Rows may be grouped by protocolidentifier (e.g., up to 8 rows per protocol). A typical group of rows isdescribed in Table 13.

TABLE 13 Field Description Pointer to R_CTL For access to IU,information category; Pointer to F_CTL For determining role ofinitiator, target, originator, responder; Pointer to CS_CTL For accessto class of service. For example, access to class of service for avirtual transaction enables a proxy to initiate a suitable class ofservice for a nonvirtual transaction. For access to PREFERENCE bit foreffecting a policy value. Pointer to PREEMPTION bit For access toPREEMPTION bit for effecting a policy value. Pointer to TYPE Fordetermining protocol identifier, enabling access to IU data structuresof various protocols, and analysis of link service requests. Pointer toFQXID May consist of several pointers. For access to routing informationprovided in the frame. Pointer to RO Used to properly place framepayload in a cache for a proxy, cache agent, or mirror agent process.Used to determine nonvirtual LBA, page, and sector; and, whether theframe data traverses a page boundary in the nonvirtual resource. Pointerto LUN For access to a logical unit number. Pointer to OP_CODE Fordetermining type of command, type of command descriptor block. Foraccess to parametric values specified with the command. Pointer to LBAUsed to properly place frame payload in a cache for a proxy, cacheagent, or mirror agent process. Used to determine nonvirtual LBA, page,and sector; and, whether the frame data traverses a page boundary in thenonvirtual resource.

A submitter manages the presentation of queries to a memory circuit thatcontains routing information or cross reference information. Forexample, submitter 1422 (comprising combinatorial logic) receivesinformation from which a query (e.g., a flow query or subflow query) isformed as described above with reference to Table 10. The query ispresented on bus ROUTE 1166 and the reply is returned on the same bus tosubmitter 1422. If the flow query reply indicates that a subflow queryshould be made, submitter 1422 prepares a subflow query. Results of flowand subflow queries are communicated to frame processor 1424 via line1472. Frame processor 1424 may provide information for a query (e.g., avirtual flow query) to submitter 1422 via line 1474 and receive thereply via line 1472.

In one implementation, submitter 1422 includes and maintains queues forcommunicating with parser 1408, frame processor 1424, and memory circuit1162. Parser 1408 pushes an entry on each of several input queues, eachinput queue corresponding to a physical port serviced by media interfacecircuit 1406. Frame processor 1424 pushes an entry on an input queue foreach subflow lookup. When a query corresponding to an entry from aninput queue is presented to memory circuit 1162, submitter 1422 may popthe entry from the respective queue and push an entry onto an outputqueue derived from the input queue entry and the result received frommemory circuit 1162. Frame processor may pop entries from the outputqueue for analysis (e.g., when a subflow flag is asserted, when avirtual flow is indicated, when the frame is a link service request).

If the stall flag is asserted in a flow lookup result, frame processor1424 pushes the corresponding entry from the input queue onto arecirculation queue. Preferably, submitter 1422 includes a recirculationqueue for each input queue having entries enqueued by parser 1408. Bypushing an entry onto a recirculation queue, further processing of theentry by submitter 1422 will be delayed. A timer loaded from a presetvalue (e.g., a CSR) counts down a duration of the delay, lapse of whichdictates when a recirculation queue requires service. A delay may allowtime for frame processor 1424 to revise field values or supplyadditional values that may be part of a subsequent query, for example,revising D_ID to route the frame to managing process 202 for analysis,setting a PREEMPTION flag, setting a PREFERENCE flag, or supplying aresource identifier (e.g., LUN) from memory. A delay may result fromimplementing a policy value. For example, frame processor 1424 mayaccomplish traffic shaping by setting the stall flag in a CAM 1306result corresponding to a flow, subflow, or virtual flow as discussedabove. Once a stall flag is set in CAM 1306, processing of asubsequently received frame will be stopped by submitter 1422 by postingthe received frames query onto a recirculation queue. Submitter 1422 maycommunicate to parser 1408 the status of recirculation queues to enableparser 1408 to inform media interface circuit 1406 that buffer-to-buffergrants should be denied due to the stall condition.

Submitter 1422 may include an arbitrating circuit to govern queueselection for submitting entries to memory circuit 1162. In oneimplementation, submitter 1422 services queues in priority from highestto lowest as: recirculation queues, flow lookup queue, and parser inputqueues (equal in priority).

Access circuits provide an interface between a frame processor and asupervising processor for example to facilitate communication of linkservice requests and replies. For example, access circuits 1404 includesupervisor queue 1490, direct memory access controller 1491, andinterrupt logic 1492. Frame processor 1424 communicates with accesscircuits 1404 in any conventional manner indicated functionally by line1409. Supervising processor 1160 communicates with access circuits 1404via SUPRV bus 1164.

Cross reference circuits maintain associations among identifiers andother data facilitating access by a processor (e.g., a routing processoror a supervising processor) or access by circuits operating in parallelwith a processor (e.g., a parser may post entries to a descriptor orpost pointers in a frame buffer for later reference by a frameprocessor; dequeue logic may obtain other data for formatting a frame tobe sent to the fabric by using various identifiers as index values).Cross reference circuits may be implemented with any conventional memorytechnology (e.g., random access memory or content addressable memory)and any conventional data storage technology (e.g., ring buffer, indexedlist, or hierarchical data structure). Cross reference circuits may beimplemented as a central memory circuit, multiported memory circuit, oras separate independently accessible memory circuits. For example, crossreference circuits 1425 include frame buffer 1480, descriptors 1481,port table 636, context table 626, virtual context table 630, page table632, and sector table 634.

Frame buffer 1480 may be used by frame processor 1424 or supervisingprocessor 1160 to retain information relative to received frames (e.g.,for analysis of the frames), to reserve space for frames being assembledin ingress buffer 1410, or to provide space for frames prior to transferinto ingress buffer 1410. Frames in frame buffer 1480 may be accessedwith reference to a frame handle. In one implementation, parser 1408assigns a frame handle to received frames and creates a worklist queuein memory accessible for reading, modifying, and deleting by parser1408, submitter 1422, VOQC 1428, and frame processor 1424. The worklistqueue describes a frame and may serve as space for results of analysisand processing that relate to the frame. A worklist queue entry mayinclude number of slots in ingress buffer 1410 available for use by thisport, a tag to be used for a query to be submitted by submitter 1422, apointer to the respective frame in ingress buffer 1410 or in framebuffer 1480, pointers to fields of the frame or to values (e.g.,pointers) in index 1420 for access to fields of the frame, one or moreflow index values (e.g., an 8-bit identifier corresponding to an S_ID(or similar 24-bit value), to an S_ID/D_ID pair, or to an FQXID) andflags for results of analysis (e.g., virtual, nonvirtual, stalled,subflow lookup available, or link service request). A frame table listsfor each frame a starting address to locate the frame in the ingressbuffer. The frame table may be indexed by a frame identifier forconvenient reference in other tables (e.g., worklist, supervisor queue,or virtual output queue). Each row of the frame table may include valuesdescribing or limiting the purpose of storing the frame (e.g., stalenesstimestamp, reason for keeping the frame, or reason for stalling theframe).

As discussed above, portions of an ingress buffer may be used for framesreceived from frame I/O ports and other portions may be used for framesdestined to be sent or transmitted to fabric 213. Analogously, portionsof an ingress buffer may be used for data to be received by asupervising processor (copied out of an ingress buffer) and otherportions may be used for transmitting data from a supervising processor(copied into an ingress buffer). According to various aspects of thepresent invention, references to these portions may be organized toimplement ring buffers using entries in linked lists. For example, aportion of multi-purpose memory 1304 may include descriptors 1481 toimplement a transmit buffer (TXB) and a receive buffer (RXB) as in FIG.15. Multi-purpose memory 1304 includes descriptors for a transmit buffer1502 (having entries 1511 and 1512 each with starting addresses 1513 and1514 respectively) and descriptors for a receive buffer 1506 (havingentries 1531 and 1532 each with starting addresses 1533 and 1534respectively). Each descriptor includes an entry in a linked list. Eachentry includes fields for flags, target identifier, source identifier,length, and a pointer to the next entry of the list (e.g., null if thisis the last entry). Because entries in the list may be inserted ordeleted regardless of position in the list and because a search of thelist may begin with any item and proceed forwards from the last item tothe first or backwards from the first to the last, the linked list canbe used as a ring buffer. For communication with a supervisingprocessor, fields have contents as described in Table 14.

TABLE 14 Field of Entry of a Descriptor Description Flags Values set orcleared to facilitate search of the linked list. Suitable values mayindicate “sent”, “acknowledged”, and/or processing “complete”. Listmaintenance may be facilitated by reading flag values (e.g., indicatingthat a list entry may be reused for another purpose, or freeing the listentry to permit use of memory for any purpose). Target A start addressfor content to be written by DMA controller 1491. For a transmit bufferentry (e.g., 1511) the target address is typically an address of ingressbuffer 1510 (e.g., payload 1521 of TXB 1504). For a receive buffer entry(e.g., 1531) the target address is typically an address of data memory1208. Source A start address for content to be read by DMA controller1491. For a transmit buffer entry (e.g., 1511) the source address istypically an address of data memory 1208. For a receive buffer entry(e.g., 1531) the source address is typically an address of ingressbuffer 1510 (e.g., payload 1541 of RXB 1508). Length The number of bytes(or other suitable measure) to be copied by the read or write operationof DMA controller 1491. Next A pointer to the next list entry or null ifnone. A tail pointer value may be set to the start address of an entryhaving a null NEXT field value (e.g., 1512 and 1532). A head pointervalue may be set to the start address of a first entry in the list. Forexample, a TXB head pointer may specify address 1513; and the value ofNEXT in entry 1511 identifies the starting address 1514 of entry 1512.An RXB head pointer may specify address 1533; and the value of NEXT inentry 1531 identifies the starting address 1534 of entry 1532.

DMA controller 1491 may refer to a head and a tail pointer for eachbuffer (TXB and RXB). Head and tail pointers may be specified bysupervising processor 1160 when setting up a DMA transfer. Head and tailpointers may refer to the first and last entry in the buffer. When anentry has been copied by DMA controller 1491, DMA controller 1491 mayrevise the head pointer to the value of the NEXT field in the entry thathas been completed. When DMA controller 1491 detects that a tail pointerfor any buffer is not equal to the head pointer for that buffer, DMAcontroller 1491 may perform the specified copy operations (e.g., oneoperation per entry) until the head and tail pointers are equal.

To simplify maintaining multiple CAM associated data records that referto a router port identifier (or virtual output queue), CAM associateddata includes logical port identifiers. The conversion of a logical portidentifier to one or more physical port identifiers is accomplished byframe processor 1424, VOQC 1428, or egress buffer 1414 with reference toa port table. A port table 636 entry may include fields as described inTable 15. Multiple entries for the same logical port identifier may beused to specify a broadcast or multi-cast routing.

TABLE 15 Field of Port Table 636 Description Logical An indirectreference to a physical port of the router. router port identifierPhysical An identifier of an output physical port of the router. routerport May be a frame I/O port (e.g., an N-port, an E-port, identifier ora port to managing processor 1112 for communication with a processperformed by the managing processor). This field may identify a trafficclass and output queue emptied via a physical router port to network101.

A context table provides storage for port identifiers and policy valuesto be associated with all communication (e.g., sequences) of thetransaction. A context table 626 entry may include fields as describedin Table 16.

TABLE 16 Field of Context Table 626 Description Transaction Anidentifier assigned by the requester or a proxy acting as a requester.For a identifier SCSI over Fibre channel protocol, the requester may bean initiator and the transaction identifier may be the value of theOX_ID field. Source port An identifier of the port from which a requestof the transaction originated (e.g., identifier an initiator's portidentifier or network address). The source port identifier may be thevalue of the S_ID field in a received frame. Destination An identifierof the port to which a request of the transaction is directed (e.g., aport identifier participant's port, a target port identifier, or networkaddress). The destination port identifier may be the value of the D_IDfield in a received frame. QoS Policy values to use for this transaction(e.g., including specification of traffic class). The QoS as enteredinto the context table may be copied from or derived from field valuesof the received frame including CS_CTL. The QoS value may be derivedfrom routing information including traffic class that has beenassociated with this requester (e.g., S_ID field value) or a participant(e.g., D_ID field value). Logical Specifies a route in accordance withone or more rows of port table 636. The output port value for logicaloutput port may be determined in accordance with the value of the S_IDfield, the physical input port of the router that received the frame,and/or the traffic class. When routing information defines severaloutput ports and/or traffic classes, the routing processor determinesone output port and traffic class (e.g., using conventional methods suchas shortest path) and may specify the virtual output queue here toimplement that route. Statistics An identifier of a counter to beconditionally incremented when a frame of this counter transaction isprocessed. Whether or not to increment the counter may depend identifieron whether a time of day (or a portion of a time value) falls within arange defining a period for collecting statistics. Flags A compositevalue that may include: a value to use in place of CS_CTL (e.g., forimplementing a policy value); and/or a value that indicates one or moreof the following: whether this transaction is stalled, or whether todrop frames of this transaction (e.g., set in response to a link servicerequest to cancel the task or transaction, or set to implement asecurity function).

A virtual transaction is a transaction recognized by the router asreferring to a virtual participant (e.g., a virtual member, virtualresource, or portion of a virtual resource). For a SCSI over Fibrechannel protocol, a transaction is recognized as a virtual transactionwhen: (a) a frame has a value in the D_ID field that is recognized (fromrouting information or a predetermined range of values) as a virtualmember; (b) a frame has a value in the LUN field that is recognized(from routing information or a predetermined range of values) as avirtual resource (e.g., storage or process); or (c) a frame has a valuein an address field that is recognized (from routing information) as avirtual address (e.g., virtual LBA for storage, virtual page (such aspart of an LBA), virtual sector (such as part of an LBA), or an objectreference).

In one implementation, the D_ID field provides a composite valueincluding an identifier of the router (e.g., a domain), and anidentifier of a virtual member. For example, a 24-bit D_ID field [23:0]having bit 15 zero provides a nonvirtual destination port identifier inbits [14:8] and a loop port identifier (e.g., AL_PA) in bits [7:0]. AD_ID field having bit 15 set provides a virtual destination portidentifier in bits [7:0]; and, bits [14:8] may be used for routing onfabric 213.

A virtual context table 630 entry may include fields as described inTable 17.

TABLE 17 Field of Virtual Context Table 630 Description Requester Anidentifier of the requester that originated the virtual transaction. Forexample, for a FCP_CMND frame of a virtual transaction from aninitiator, the value of the S_ID field. When the router originatesframes of the virtual transaction back to the initiator (e.g., inresponse to frames received from a nonvirtual target), the value of theD_ID field in the frame originated back to the initiator will beassigned by the router from the Requester entry in this row of thevirtual context table. Virtual An identifier assigned by the requesterthat identifies this virtual transaction. For transaction example, thevalue of the OX_ID field. The virtual transaction identifier is usedidentifier as an index into this virtual context table 630 to obtain oneor more rows of this virtual context table 630. Virtual An identifierassigned by the router for use in a virtual transaction (e.g.,participant's VRX_ID). transaction identifier Nonvirtual An identifier(e.g., determined from routing information) of a nonvirtual participantparticipant of a nonvirtual transaction used to implement an intent ofthe virtual transaction. May be used as the value of an D_ID field in aframe of such a nonvirtual transaction (e.g., directed to the nonvirtualtarget corresponding to the virtual target of the virtual transaction).Nonvirtual An identifier of a nonvirtual transaction to be conducted toaccomplish an intent transaction of a virtual transaction. Typicallyassigned by a routing processor (e.g., acting as identifier requester,initiator, or originator). Typically assigned when the virtualtransaction is to be routed (e.g., the first routing processor betweenan initiator and a target having routing information sufficient for thatrouting processor to recognize the transaction as a virtualtransaction). The nonvirtual transaction identifier (e.g. NVOX_ID) maybe used as the value of the OX_ID field of a frame of a nonvirtualtransaction. The nonvirtual transaction identifier is used as an indexinto this virtual context table 630 to obtain one or more rows of thisvirtual context table 630. Nonvirtual An identifier assigned by aparticipant of a nonvirtual transaction (e.g., participant's NVRX_ID).The participant's transaction identifier may be determined from atransaction frame directed back to an initiator (e.g an RX_ID field of aresponse). identifier Nonvirtual An identifier of a portion of anonvirtual transaction. A SCSI I/O comprising sequence sequencescorresponds to a transaction. Nonvirtual sequence identifier may beidentifier assigned by a proxy or a routing processor (e.g., NVSEQ_ID).Nonvirtual An identifier of a relative offset that describes the offsetinto a buffer of the data offset being conveyed by the payload of thisframe. For example, a virtual transaction may be described by a relativeoffset (e.g., the value of field RO in a FCP_DATA frame). Thecorresponding nonvirtual transaction may be conducted independently ofthe virtual transaction (e.g., in a delivery order specified by thenonvirtual target). A payload in the nonvirtual transaction maytherefore be identified by a nonvirtual offset (e.g., NVRO). Flags Acomposite value that may include whether to discard the frame (e.g., toenforce access control or abort a task), whether to route the frame to aproxy process, or whether to pass the frame to a supervising process.Timestamp A value indicating time when this row was created. Used toflush stale rows of this virtual context table 630.

According to various aspects of the present invention, a virtualidentifier is associated with a nonvirtual entity so that reference tothe virtual identifier accomplish communication affecting the nonvirtualentity. This association may be implemented as a virtual memberidentifier associated with a nonvirtual member identifier (e.g., amember-to-member association). Alternately, the association may beimplemented with member/resource-to-member/resource association, amember/resource/address-to-member/resource/address association, amember/object_reference-to-member/object-reference association, orpermutations of these. These associations may be one to many (e.g.,facilitating redundancy in storage or processing). When a storageresource address includes further segmentation, (e.g., a logical blockaddress may include a page, sector, and block offset), a reference to avirtual sector may affect one or more nonvirtual sectors.

In one implementation, storage virtualization is implemented using oneor more page tables and one or more sector tables. For example, eachvirtual resource identifier may be associated with one page table thatincludes one or more rows. Each page table row is associated with onesector table that includes one or more rows.

In an alternate implementation, a query for routing information mayresult in a maximal match of the query tag that includes in order:member identifier, resource identifier, page address, sector address,and block address. In a preferred implementation, described below, theblock address is omitted and storage virtualization is accomplished tothe sector level.

A page table stores a one-to-many association between an identifier of avirtual storage address and a nonvirtual storage address. A page table632 entry may include fields as described in Table 18. When routinginformation provides a direct reference (as opposed to an indexedreference) to a particular page table, the virtual resource identifierfield may be omitted.

TABLE 18 Field of Page Table 632 Description Virtual An identifier of avirtual resource for which a corresponding nonvirtual page resourcetable has been defined. identifier Virtual Page An identifier of a pageas referred to in a virtual transaction. For example, a Address CDB of avirtual transaction may include a value in the LBA field comprisingvirtual page address (as well as a sector and a block address). VirtualAn identifier of the first sector of a list of sectors described in asector table Sector List describing nonvirtual sectors that implementthe virtual page. Each sector of the list may be located on a differentnonvirtual resource and/or at different pages of a nonvirtual resource.Valid A flag indicating whether this row of page table 632 is valid.Permits efficient reuse of a row.

A sector table stores an ordered list of sectors that comprise a page.For example, if a virtual page comprises 512 sectors, then a sector listassociated with that virtual page (e.g., a row of a page table,discussed above) includes 512 rows, the first row corresponding to thefirst sector of the virtual page, and so on. A sector table 634 entrymay include fields as described in Table 19. The virtual sector addressfield may be omitted from sector table 634 when the order of sectors ismaintained by design (e.g., sequential order 0-511). The nonvirtualsector address field may be omitted when access by nonvirtual sectoraddress is not desired.

TABLE 19 Field of Sector Table 634 Description Virtual sector Thevirtual sector address of a virtual page. May be used as an index toobtain address associated values from this row of sector table 634.Nonvirtual The nonvirtual sector address associated with the virtualsector address of a sector virtual page. May be used as an index toobtain associated values from this row address of sector table 634.Nonvirtual An identifier of the nonvirtual member implementing thisnonvirtual sector. For member example, a value (e.g., NVD_ID) used inthe D_ID field or a nonvirtual identifier transaction. Nonvirtual Anidentifier of the nonvirtual resource implementing this nonvirtualsector. For resource example, a value (e.g., NVLUN) used in the LUNfield of a nonvirtual identifier transaction. Nonvirtual An address thatidentifies nonvirtual data for this virtual sector. For example, aaddress value (e.g., NVLBA) used in the LBA field of a nonvirtualtransaction. Nonvirtual The number of sectors (e.g., starting sectornumber, ending sector number, bounds and/or quantity of sectors) in thenonvirtual LBA. May differ from the number of sectors in the virtualLBA. May be used to determine whether a virtual transaction will cross apage boundary. Control A composite value indicating: whether thenonvirtual sector is part of a snapshot, part of a mirror, or isassociated with a cache. Routes Routing information (or one or morepointers to routing information) describing alternate paths to thenonvirtual sector. Each alternate route may designate an output queueand traffic class.

An output queue presents frames to the fabric. For supporting multipleoutput queues, a frame may be copied to a region of memory designatedwith a suitable priority and/or traffic class. Alternatively, aso-called virtual output queue may include pointers to the frame as itmay already exist in a memory that serves a function different from anoutput queue. For example, output queue 1437 includes virtual outputqueue controller (VOQC) 1428 and dequeue logic 1412. Output queue 1437refers to the frame as it exists in ingress buffer 1410, therebyavoiding the time and resources needed to maintain a copy of the framein a memory different from the ingress buffer. Dequeue logic 1412(comprising combinatorial logic) presents frames to distributing circuit1402 via line 1434 (or portions of frames such as identifiers andpayloads as discussed above with reference to distributing circuit1402). Dequeue logic 1412 is directed by command signals received fromVOQC 1428 via line 1442. Commands include directives to format datapointed to by pointers maintained by VOQC 1428 to form frames asdirected, send frames, drop frames (e.g., to interrupt a link inresponse to a suitable link service request or exceptional condition),and stall queues (e.g., as discussed above with reference to FIG. 9).

VOQC 1428 may include combinatorial logic and/or one or more statemachines to perform methods discussed above with reference to FIGS. 9and 10 for each output queue. VOQC also receives flow control signalsfrom distributing circuit 1402 via line 1436. If operation ofarbitrating circuit 1405 and egress buffer 1414 result in a buffer fullbeyond a threshold, egress buffer 1414 provides flow control signals todistributing circuit 1402 via line 1432, as discussed above.Distributing circuit may respond to such flow control signals and toexceptional conditions (e.g., loss of synchronization or timing delayassociated with receiving signal RING-I or providing signal RING-O, lackof sufficient grants for sending, initialization, reinitialization, orhigh error rates) by asserting flow control signals to VOQC 1428 vialine 1436.

Output queues may be implemented as ring buffers in ingress buffer 1410.When permitted by flow control signals 1436 and according to a method ofarbitrating among sources of similar priority (e.g., as discussedabove), VOQC 1428 identifies to dequeue logic 1412 via line 1442 datafor sending to the fabric from a ring buffer in ingress buffer 1410.Upon successful processing of the identified frame, dequeue logic 1412may adjust the ring buffer pointers to remove the identified frame fromthe queue. A region of memory removed from a ring buffer may bereallocated to any other function provided by ingress buffer 1410. Byallowing reallocation of ingress buffer memory, frames of varying lengthmay be accommodated, queues of varying capacity may be accommodated,queue stalls may be implemented by allowing a ring buffer to grow insize, flow controls such as buffer grants may be implemented, andarbitration may be accomplished based on information associated witheach output queue including identifiers related to flow, subflow,virtual flow, destination, protocol, resource, traffic class, andpriority.

The number of virtual output queues maintained by VOQC may includemultiple queues for the same destination output physical portidentifier. For example, in one implementation, a frame to be enqueuedfor output to fabric 213 is pushed into a queue corresponding to thesource physical port identifier from which the frame was received bythis port logic circuit (e.g., a port logic circuit may serve 4 portsfor input), and further corresponding to a traffic class. Consequently,an arbitrating circuit for one physical output port arbitrates among alarge number of queues. For example, when a port logic circuit serves 4physical input ports (local to this port logic circuit), recognizes 4traffic classes, and routes frames on fabric 213 to up to 20 physicaloutput ports (e.g., 16 frame I/O ports in router 102, one port to amanaging process (e.g., 204), and one multicast port), VOQC manages atotal of 320 queues. Each of 20 arbitrating circuit selects from 16queues.

An entry in a virtual output queue may include the values described inTable 20.

TABLE 20 Field of virtual output queue entry Description Frame to outputpointer to a frame in ingress buffer 1410; flag for CRC regeneration;Principal output queue physical or logical output port identifier; portspeed to be used for the output port; priority (e.g., if set,arbitration according to traffic class may be superceded); Multicastmulticast destination identifier (e.g., D_ID); Statistics statisticscounter identifier; Secondary output queue physical or logical outputport identifier; port speed to be used for the output port; priority(e.g., if set, arbitration according to traffic class may besuperceded); Miscellaneous midswitch stage identifier;

The secondary queue, if specified, may direct that a copy of the framebe sent to a managing process 204, to an administrating process 202, orto another resource. The managing process, administrating process orresource may appear (if not intentionally made invisible) as a target(e.g., a virtual target) to the source of the frames. Sending a copy ofa frame to a managing process may facilitate configuration management byprocess 406, or report generation by process 410 (e.g., accumulation oftraffic statistics in addition to statistics reported by routers102-105). Sending a copy of a frame to an administrating process mayfacilitate monitoring for security purposes. Frames to be sent to asecondary output queue may be selected based on traffic statistics, oron type of frame. For example, frames writing a primary data store (notreading the primary data store) would be sent to a mirror data store.Sending frames to a managing processor that hosts a mirror agent (426)facilitates receiving frames in a first order preferred for the firstinitiator; and acting as an initiator for serving the mirror resource ina second order preferred for operation of the mirror resource. Themirror agent may perform initiator and data transfer functions analogousto a proxy process, discussed above.

A statistics store accumulates counts of traffic statistics as describedabove. For example statistics store 1426 (comprising combinatorial logicsuch as counters, and memory) receives specifications for whatstatistics to accumulate, how to accumulate them, and provides statusand results to frame processor 1424 via line 1460. Statistics store 1426receives notice of events for possible counting from media interfacecircuit 1406, arbitrating circuit 1405, parser 1408, dequeue logic 1412,and virtual output queue controller 1428. Counts may be restricted toevents related to a particular subflow of a link and/or to a particularprotocol used on a link. For example, counts may be accumulated forframes exceeding a threshold length that are received on a particularphysical port, are directed to a particular subnetwork resource (e.g.,process or device), and contain indicia of a particular upper levelprotocol (e.g., CORBA, SMTP, or VI), upper level file system (e.g.,UNIX, WINDOWS), or upper level file type (e.g., images such as jog,movies such as .mpeg, audio such as .wav, text such as .doc, databasesuch as .index). Counts may be accumulated for intervals as directed byframe processor 1424. Intervals may be specified at random, of randomduration, or at particular times and particular durations. The starttime for collecting a specified statistic may also be set in accordancewith the occurrence of an event (e.g., another statistic counter hasexceeded a threshold).

Statistics counters may cooperate with values recalled from CAM 1306 todetermine whether an event should be tallied. For example, CAM 1306 mayprovide one or more values that specify a sampling window (e.g., hourly,daily, weekly; with a relative start time (8:15 a.m. each day),duration, and/or relative end time (8:30 a.m. each day)) during which anevent should be tallied or a frame sampled for purpose of determiningwhether a countable event is indicated by the contents of the frame.Current time of day may be compared to the values specifying thesampling window to determine whether to ignore the frame, count it(e.g., as a unit or accumulate its length), or analyze it for possiblestatistics.

A routing processor includes any stored program computer circuit and/orstate machine that performs a routing process 208 as described abovewith reference to FIG. 2. A routing processor 1161 may include a portlogic circuit and interfaces to a supervising processor and to a memorycircuit. Alternately, a routing processor 1161 may further include amemory circuit or exclusive use of a portion of a memory circuit. Forexample frame processor 1424 in one implementation performs methodsdescribed above with reference to FIGS. 2 and 6, and portions of FIGS.7-10 that are not implemented in separate circuits for parallelprocessing. Frame processor 1424 includes EPM to enable downloadingprograms to be executed by routing engines. Such programs may differamong routing engines of router 201, for example, to implement differentprotocol support on different frame I/O ports or at different timesduring operation of the same frame I/O port. Frame processor 1424 hasaccess to shared memory 1162 (e.g., for read, write, fetch, indirectaddressing, stacks, and heaps) via ROUTE bus 1166.

To process a link service request received from any frame I/O port,parser 1408 indicates to frame processor 1424 that a link servicerequest has been placed in ingress buffer 1410 by signals on line 1468.Alternatively, frame processor 1424 may determine that a frame receivedis a link service request by reading portions of ingress buffer 1410identified by index 1420. Frame processor 1424 may create an entry insupervisor queue 1490 to notify supervising processor 1160 of thereceived link service request. Supervising processor 1160 may respond toan interrupt generated by interrupt logic 1492 or may periodically readsupervisory queue 1490 to discover the pending queue entry. Supervisingprocessor 1160 may then set up DMA controller 1491 and interrupt logic1492 to copy suitable portions of the frame from ingress buffer 1410 todata memory 1208 for convenient access during processing of the linkservice request. The region of ingress buffer 1410 used by the frame maythen be freed for other use as discussed above. The entry in supervisorqueue 1491 may also be freed.

Supervisor queue 1490 may be implemented as a ring buffer or array inany suitable memory circuit. For example, memory for supervisor queue1490 may be allocated from a portion of ingress buffer 1410 or frommulti-purpose memory 1304 (e.g., a reserved region or a region shared-bymultiple frame processors). When placing entries in supervisor queue1490, a priority value may be associated with the entry so thatprocessing by supervising processor 1160 may respond to higher prioritylink service requests that enter the queue in time after lower prioritylink service requests. Priority may be determined with reference to oneor more of the following: identifiers of a routing processor, a type oflink service request, a flow, a subflow, a virtual flow, a resource, ora protocol.

To process one or more link service replies to be sent to any frame I/Oport(s), a managing process, and/or an administrating process,supervising processor 1160 may set up DMA controller 1491 and interruptlogic 1492 to copy suitable portions of one or more frames from datamemory 1208 to ingress buffer 1410. If any frame written by DMA isplaced in a region of ingress buffer 1410 monitored by frame processor1424, frame processor 1424 may act on the frame in any manner discussedabove (e.g., to accomplish routing a flow, subflow, or virtual flow) Ifany frame written by DMA is placed in an output queue managed by VOQC1428, the frame is passed to the fabric as discussed above whereupon theframe is received from the fabric by a suitable egress buffer anddelivered via a frame I/O port as discussed above. Supervising processor1160 may respond to an interrupt generated by interrupt logic 1492 ormay periodically read supervisory queue 1490 to discover completion ofprocessing of frames to facilitate subsequent DMA of further frames asdesired.

As discussed above, a transaction (e.g., an input/output (I/O) or anexchange) for accomplishing a data transfer between members may includeeither a read operation (the subject data being from the participant) ora write operation (the subject data being to the participant). Such atransaction is generally referred to as a R/W I/O (e.g., the termstransaction, I/O, and exchange, referring loosely and generally toeither a read or a write operation). According to Fibre Channel and SCSIprotocols, a R/W I/O includes transactions between an initiator and atarget that employ frames (or IUs) identifiable as FCP_CMND,FCP_XFER_RDY, FCP_DATA, and FCP_RSP. According to various aspects of thepresent invention, a routing processor implements methods of routing R/WI/Os for nonvirtual and for virtual transactions. For example, frameprocessor 1424 detects frames of nonvirtual non-R/W I/Os and identifiesthem for processing by supervising processor 1160. Frame processor 1424detects frames of virtual non-R/W I/Os and routes them for processing bya proxy process of managing processor 1112.

A method of processing frames according to various aspects of thepresent invention may include any suitable combination of the followingoperations: receiving a frame from a network; determining in a routingprocessor whether the received frame is a data frame (e.g., part of adata transaction); if the received frame is not a data frame (e.g., partof a control transaction), identifying the frame for processing (e.g.,by a supervising processor for nonvirtual control-frames; and by a proxyprocess for virtual control frames); if the received frame is a dataframe, determining a resource identifier referred to by a R/W operationof the data transaction; determining a nexus (e.g., an I_T_L nexus orI_T_L_Q nexus) of the R/W operation, recalling a policy value associatedwith the nexus; enqueueing data for the R/W operation in a first bufferin accordance with the policy value; dequeueing data from the firstbuffer for transfer on a fabric by arbitrating among queues inaccordance with a historical value; adjusting the historical value inaccordance with the amount of data transferred on the fabric; receivingdata from the fabric; enqueueing data received from the fabric in asecond buffer; and dequeueing data from the second buffer for transferto the network. Such a method may be performed by the cooperation of arouting processor, managing processor, and supervising processor. Forexample, login, proxy, and error condition handling operations may beaccomplished by managing processor 1112 in cooperation with a routingprocessor and supervising processor, as discussed above.

In the following discussion, a SCSI I/O R/W sequence is an example of aR/W I/O series; other SCSI sequences are examples of a non-R/W I/Oseries.

For example, a series of messages 1600 of FIG. 16 includes a non-R/W I/Oseries 1601, a nonvirtual R/W I/O series 1621, and a virtual R/W I/Oseries 1631. In the non-R/W I/O series 1601, an initiator (e.g., anonvirtual member of network 101) 1660 sends at time 1602 a message “A”(e.g., a link service request). Message “A” is addressed to a router(e.g., router 102) using a well known address. Routing processor 1661(e.g., having a port logic circuit 1186 for a suitable number of ports,a memory circuit 1162, supervisor bus 1164, and route bus 1166)cooperates with supervising processor 1160 as described for link servicerequests in Table 23, e.g., sending message “B” at time 1604. When alink service request affects the state of a proxy or requiresinformation maintained by a managing process as discussed above,supervising processor 1160 sends at time 1606 a message “C” to managingprocessor and receives a response message “D” at time 1608. Messages “C”and “D” are conveyed by LAN 210, 1132. Supervising processor 1160, onreceipt of status or information from message “D”, sends at time 1612message “E” to routing processor 1661. A response message “F” is sent byrouting processor 1661 to initiator 1660 at time 1612. Messages “B” and“E” are conveyed by bus 212, 1164. Messages “A” and “F” are conveyedthrough fabric 213, 1170, 1402.

In the nonvirtual R/W I/O series 1621, no messages refer to virtualports. Initiator 1660 sends at time 1622 a message “G” (e.g., anFCP_CMND to read data from a target) that is routed by routing processor1661 as message “H”, sent at time 1624 to target 1662. Target 1662responds by sending at time 1626 message “I” that is routed by routingprocessor 1661 as message “J” sent at time 1628 to initiator 1660. Whenthe non-R/W I/O is one sequence of a transaction, the remainingsequences (e.g., FCP_XFER_RDYs, FCP_DATAs, and FCP_RSP, andacknowledgement frames) would follow the same processing path G-H-I-J.As discussed above, architecture 200 of router 201 accomplishesnon-blocking routing for, inter alia, nonvirtual R/W I/Os.

In the virtual R/W I/O series 1631, messages “K” and “P” refer to avirtual target as opposed to nonvirtual target 1662. Initiator 1660 attime 1632 sends message “K” addressed to a virtual target (not shown).If routing processor 1661 routes the virtual R/W I/O message “K” at time1634 to a proxy process 418 using a well known address of the proxyprocess, the proxy process (hosted by managing processor 1112) at time1636 responds with message “M”. Routing processor 1661 may route message“M” as message “N” addressed from the proxy process to a nonvirtualtarget 1662 at time 1638. A message “M” may be addressed to initiator1660 and so be routed back to initiator 1660 as message “P”. On theother hand, when no assistance from proxy process is desired, routingprocessor 1661 may respond to message “K” by sending message “N”addressed to target 1662 from the proxy process. When target 1662responds, it sends at time 1640 message “O” addressed to the proxyprocess. Routing processor 1661 intercepts message “O” and, withoutcommunication with the proxy process, routes message “P” at time 1642 toinitiator 1660 addressed from the virtual target (not shown). Messages“L” and “M” are nonvirtual messages 1635 referring to a suitable proxy.In the message series “K”, “N”, “O”, “P” messages “K” and “P” arevirtual and messages “N” and “O” are nonvirtual. The routing of messages“N” and “O” differs from the routing of messages “H” and “I” in that theidentifiers for source and destination in messages “G”, “H”, “I”, and“J” are unchanged during routing. By contrast, the routing of messages“N” and “O” may include saving the source and destination identifiersfrom message “K” and rewriting the source and destination identifiers toform so-called redirected messages “N” and “O” 1339 by routing processor1661.

A proxy process of managing processor 1112 acting as an initiator maysend 1629 a message (e.g., discovery, port log-in, process log-in, orSCSI commands such as RESET_UNS) to target 1662 (not shown). Target 1662may reply to such a command by sending a message (not shown) to theproxy (e.g., completion status).

Processing for representative control frames and data frames isdescribed in Table 21. FCP_CMND, FCP_XFER_RDY, and FCP_DATA sequencesrepresent data transactions having data frames. Other sequences in thetable represent control transactions having control frames.

TABLE 21 Received Frame Description of processing FLOGI (fabric login asRouting processor: identify frame (1702, 1704) as a link servicedefined, e.g., in FC-FS) request; identify frame to supervisingprocessor (1706). Supervising processor: reply to FLOGI and acceptservice parameters according to requested class of service. RTIN(request network Routing processor: identify frame as link servicerequest, identify topology information as frame to supervisingprocessor. defined, e.g., in FC-FS) Supervising processor: reply to RTINwith identifiers of members and resources. PLOGI (port login as Routingprocessor: for nonvirtual destination identifier, identify defined,e.g., in FC-FS) the frame as a link service request, identify frame tosupervising processor. For virtual destination identifier, pass theframe to the corresponding proxy in the managing processor. Supervisingprocessor: prepare suitable reply. Managing processor: initiate secondPLOGI to nonvirtual target that corresponds to the virtual targetindicated in the first PLOGI; respond to the first PLOGI in accordancewith the result of the second PLOGI from the nonvirtual target. PRLI(process login as Routing processor: for nonvirtual destinationidentifier route as defined, e.g., in FCP-2, not conventional traffic tothe nonvirtual destination. For virtual handled herein as a linkdestination identifier, pass the frame to the corresponding proxy inservice request) the managing processor. Managing processor: performoperations analogous to PLOGI, discussed above. REPORT LUNS (requestRouting processor: for nonvirtual destination identifier route as forlogical unit numbers as conventional traffic to the nonvirtualdestination. For virtual defined, e.g., in SPC-3) destinationidentifier, pass the frame to the corresponding proxy in the managingprocessor. Managing processor: reply with list of LUNs (nonvirtual andvirtual) that are permitted to be accessed by this requester; may deferstoring LUNs in CAM 1306 until first access attempt is recognized.FCP_CMND (FCP header Routing processor: for nonvirtual destinationidentifier route as with SCSI command CDB conventional traffic to thenonvirtual destination. For virtual in payload as defined, e.g.,destination port identier, prepare a forward frame to route in place inFCP-2) of the received frame as described, inter alia, in FIGS. 17-20.FCP_XFER_RDY (transfer Supervising processor: no involvement except forinitialization ready sequence with and updating maps. response data asdefined, Managing processor: no involvement except for initializationand e.g., in FCP-2 and SBC-2) updating maps. FCP_DATA (data transfer fora read or write of the target as defined, e.g., in FCP-2 and SBC-2)

A routing processor provides nonblocking routing of R/W I/Os. A method1700 of FIGS. 17-20 provides routing of non-R/W I/Os, nonvirtual R/WI/Os and virtual R/W I/Os as follows. Routing processor 1161 recognizesa frame received from network 101 via a frame I/O port as R/W I/O (1702,1704); recalls a flow lookup from CAM 1306 (1716) and if incomplete ormissing (1718), gets a supervising processor 1160 to analyze the frame,specify a route, or drop it (1706-1714). If the subflow flag is set inthe result of the flow query (1720), the routing processor does subflowlookup (1722) from CAM 1306 and reports errors to the supervisingprocessor (1724, 1706-1714) possibly stalling frame in submitter queueawaiting CAM update by supervising processor.

Context and virtual context may be stored locally (e.g., in memoryaccessible to a frame processor of a port logic circuit, for example, onthe same substrate as the frame processor), stored in multi-purposememory 1304, or stored in RAM 1312. When context table 626 and/orvirtual context table 630 are stored locally, a frame received at afirst port logic circuit is tested as to whether the routing processorhas access to context (1726); and, if context is stored elsewhere, theflow and subflow results are used to build a forward frame (1728),marked for further processing by another routing processor (1730) wherethe context is available. Otherwise, it is determined whether thecontext is already available; and, if not, a new entry for context table626 and/or virtual context table 630 is created (1734) in the localmemory or where context is stored.

Assuming that local context and/or virtual context is available, suchmay be revised with information parsed from the received frame. If a LUNis specified in the CDB (e.g., FCP_CMND frame) (1802), the routingprocessor sets subflow flag (1804) and stores modified results (1806) incontext table 626. If the frame includes an RX_ID value from the target(e.g., an ACC to FCP_CMND, or an FCP_XFER_RDY) or proxy for the virtualtarget (1808), the routing processor stores (1810) the RX_ID in contexttable (626). If there were CAM hits on the lookups, the routingprocessor tests the CAM result flag for virtual (1812). If nonvirtual,an update of the context table with tuple of S_ID, D_ID, LUN, LBA,OX_ID, and RX_ID is accomplished. The flow, subflow, and context tableare then used to route frame (1814) to an output queue (1816) per S_ID,router output port identifier and traffic class.

If the received frame is determined to be virtual (e.g., either the flowor subflow lookups indicate the D_ID is associated with a virtualtarget) and if no suitable nonvirtual transaction identifier (1902) isin virtual context table 630, the routing processor creates a tuple ofvirtual transaction identifier (e.g., original OX_ID from initiator1660) and new nonvirtual transaction identifier (NVOX_ID) and stores(1904) as a new entry in virtual context table 630. If there is no knownproxy (1912), the routing processor routes the frame to managingprocessor 1112 for analyzing, revising tables, or dropping the frame. Ifthe frame is virtual and NVOX_ID is available from virtual context table630 (e.g., from FCP_XFER_RDY, or ACK), then OX_ID, LBA and RO may beused as an index to the virtual context, page, and sector tables todetermine nonvirtual destination (NVD_ID), nonvirtual initiator (e.g., aproxy NVS_ID), nonvirtual NVLBA, and whether the amount of data to beread or written will cross a page boundary (1906, 1908). If no pageboundary will be crossed, the routing processor modifies (1910) theframe to appear as sent from a proxy in a nonvirtual transaction to thenonvirtual target.

For both nonvirtual and virtual processing, after a frame for the fabrichas been prepared (1738, 1910, or 1914), the routing processor enqueuesthe frame (1816) to the fabric and later this or another routingprocessor receives the frame from the fabric (2002). If the framereceived from the fabric is marked (1730, 2004) as requiring applicationof virtual context at this routing processor, context and virtualcontext tables are used to modify (2006) the frame to appear to havebeen sent by the virtual target to the initiator; else the frame issimply passed (2008) to the output port as for a message (“F”, “H”, “J”,“L”, “N” or “P”) directed to a nonvirtual destination.

A supervising processor cooperates with a routing processor as follows.If the supervising processor is passed a frame for which a CAM hit ismissing, the supervising processor uses the S_ID to get an ACL. If avalue for D_ID and LUN are in the ACL (possibly not in CAM because noprior access attempt), the supervising processor updates the appropriateCAM with LUN; else, if D_ID and LUN are not in the ACL, the supervisingprocessor drops the frame, implementing security of access.

A fabric according to various aspects of the present invention providesfull-mesh communication using point to point connections. Nodes of thefabric are joined by point to point connections in a topology similar insome ways to a star and in a physical arrangement similar in some waysto a ring. Each node provides a slice of the fabric circuitry. Accordingto various aspects of the present invention, a slice capable of beinginserted into a ring coupling a maximum number of frame I/O ports may beused in a ring of any lesser number of frame I/O ports, eliminatingcostly development of fabric circuits for different routers each havinga different number of frame I/O ports. The fabric may include a printedcircuit layout that need not be revised for production of various modelsof routers having support for different numbers of frame I/O ports. Animplementation of a fabric according to various aspects of the presentinvention may have any maximum number of nodes limited perhaps bytransmission delays and timing differences that may develop betweennodes. Each segment of a fabric may comprise a point to pointtransmission line driven by one transmitter and terminated by onereceiver with suitable impedance matching termination circuitry.

In one implementation of fabric 213, for example fabric 2100 of FIG. 21,full mesh communication is provided between all ports at each of fivefabric nodes. Fabric 2100 includes five circuits 2101-2105 (one at eachfabric node), each having port I/O circuitry (PIOC) that may be similarin some respects to circuitry described above with reference to portlogic circuit 1186. Each PIOC provides an interface to a plurality offrame I/O ports (not shown). In the simplified functional block diagramrepresentation of fabric 2100 in FIG. 21, a frame sent to the fabricfrom PIOC 2111 at node 2101 is coupled to node 2102 by segment 2121, isthen coupled to node 2103 by segment 2122, is then coupled to node 2104by segment 2123, and is then coupled to node 2105 by segment 2124. Inother words, fabric circuits (e.g., 2001) in cooperation with couplingsegments (e.g., 2021) at each node: (a) couple signals received from thenode one position counter-clockwise (e.g., M1 for minus one) to the paththat extends three segments clockwise (e.g., P3 for plus three); (b)couple signals received from the node two positions counter-clockwise(M2) to the path that extends two segments clockwise (P2); (c) couplesignals received from the node three positions counter-clockwise (M3) tothe path that extends one segment clockwise (P1); (d) couple signalsreceived from the node four positions counter-clockwise (M4) to the PIOCat this node; and (e) couple the signal provided by the PIOC at thisnode to the path that extends three four segments clockwise (P4).

In another implementation of fabric 213, fabric 2200 of FIG. 22 providesfull mesh communication between all ports at each of three fabric nodes.Fabric 2200 includes three circuits 2201-2203 (one at each fabric node),each having port I/O circuitry (PIOC) that may be identical to the PIOCsdiscussed with reference to FIG. 21 except that fabric circuits (e.g.,2201) in cooperation with coupling segments at each node: (a) couplesignals received from the node one position counter-clockwise (M1) tothe path that extends one segment clockwise (P1); (b) couple signalsreceived from the node two positions counter-clockwise (M2) to the PIOCat this node; and (c) couple the signal provided by the PIOC at thisnode to the path that extends two segments clockwise (P2).

The same physical printed circuit layout (not shown) may be used forboth fabrics 2100 and 2200. In fabric 2200 segments may be connected byfillers 2204-2205 across unfilled fabric node positions. Fabric 2200 maybe upgraded to fabric 2100 by replacing fillers 2204-2205 with fabriccircuits 2104-2105 and reconfiguring switching functions of fabriccircuits 2201-2203 to provide the functions of fabric circuits2101-2103. Such reconfiguration is preferably accomplished by inputs toeach fabric circuit; each fabric circuit being of an identical typehaving internal configuration functions responsive to these inputs.

Router 102 may include a fabric of the type described above withreference to FIGS. 21 and 22. In one implementation, each port logiccircuit (e.g., 1186, 1188) includes a distributing circuit havingsegment signal switching functions as discussed above. For example,distributing circuit 1402 of FIGS. 14 and 23, provides suitable couplingthrough port logic circuit 1186 so that port logic circuit 1186 may beinstalled in any position of a fabric having any number of populatedpositions (up to a predetermined maximum number of positions).Distribution circuit 1402 includes controller 2301, receivers 2305,interconnecting switch 2306, transmitters 2308, scrambler 2322,descrambler 2324, normalizing switch 2310, and back pressure logic 2312.

A controller establishes a switch configuration and segment terminationsuitable for a particular under population and total number of fabricnodes of the fabric. For example, controller 2301 (comprisingcombinatorial logic) receives TOTAL_NODES signal 2302, UNDER_POPULATIONsignal 2303 that indicates the number of positions counter clockwise ofthe present position that are occupied by fillers, and POSITION signal2304 that identifies which fabric node is associated with thisdistributing circuit (e.g., 2101, or 2102, or 2103, and so on).TOTAL_NODES, UNDER_POPULATION, and POSITION signals may each comprise abinary value having several known logic levels each provided through aprinted circuit board trace, a jumper, a manual switch, or an EPM orother memory output. A controller 2301 that receives a non-zero valuefrom the UNDER_POPULATION signal 2303 directs receivers 2305 to use asuitable impedance matching termination circuit. Controller 2302operates interconnecting switch 2306, operates normalizing switch 2310,and configures back pressure logic 2312 in accordance with TOTAL_NODESsignal 2302 and POSITION signal 2304. Typically, operations of switches2306 and 2310 and configuration of backpressure logic 2312 occur duringinitialization of router 102 and initial settings are not changed duringnormal operation of router 102.

Receivers 2305 include an independent receiver circuit for each segment.In other words, each segment is a point to point conductor with nobranches so as to simplify high frequency tuning of the conductor andmatching of one transmitter to one receiver for each segment. Receiversreceive signals RING-IN 1170 from segments of the fabric. For example,signal M1 is received by a first receiver, signal M2 by a secondreceiver, and so on. Each receiver may include a phase locked loop forclock and data recovery from the signal received from a segment.Demodulation of the signal received from a segment may include anyconventional demodulation technique (e.g., demodulation of phase shiftkeying).

Prior to transmission, data to be transmitted may be scrambled so thatenergy conveyed by the transmitted signal is distributed amongfrequencies and/or frequency bands. By distributing transmitted energy,noise immunity of the fabric is improved and noise radiation by thefabric is easier to control. Scrambler 2322 provides a scrambled signalon line 2323 in accordance with DATA signal 1434 from dequeue logic 1412associated with this distributing circuit 1402.

Interconnecting switch 2306 couples each selected signal of signals 2307to a suitable transmitter 2308. Signal selection and coupling isaccomplished in accordance with control signals received from controller2301 and in accordance with the fabric architecture discussed above withreference to FIGS. 21 and 22. Signals on lines 2307 include demodulatedsignals from receivers 2305 and the scrambled data signal on line 2323.Switch output signals on lines 2309 are coupled to transmitters 2308.

Transmitters 2308 provide signals RING-O 1172 to segments of the fabric.Transmitters 2308 include an independent transmitter circuit for eachsegment. For example, signal M1 of signal group 2309 is transmitted by afirst transmitter to provide signal P3 for a first segment, signal M2 ofsignal group 2309 is transmitted by a second transmitter to providesignal P2 for a second segment, and so on. Each transmitter may includeclock generation circuitry to train the corresponding receiver.Modulation of the signal to be transmitted on a segment may include anyconventional modulation technique (e.g., phase shift keying).

Descrambler 2324 accepts signals from receivers 2305 and independentlydescrambles each signal to provide corresponding clear data signals onlines 2311. Signals on lines 2311 are provided to normalizing switch2310 and to back pressure logic 2312.

Normalizing switch 2310 provides outputs A-E on data lines 1430 toegress buffer 1414 associated with this distributing circuit 1402. Therouting of signals received on normalizing switch inputs 0-4 to outputA-E is directed by controller 2301 so that adjustments (if any) inrouting methods performed by frame processor 1424 to account fordifferences in the installed position of port logic circuit 1186 or thetotal number of fabric nodes are simplified.

Back pressure logic 2312 receives clear data signals 2311 that mayinclude back pressure messages transmitted in response to status of anegress buffer coupled to any fabric node (e.g., from any port logiccircuit of routing circuits 1150-1152). In addition, back pressure logicmay receive CONTROL signals on line 1432 from egress buffer 1414associated with this distributing circuit 1402. Back pressure logic 2312provides CONTROL signals on line 1436 to VOQC 1428 associated with thisdistributing circuit 1402. VOQC responds to CONTROL signals on line 1436to stall or restart any one or more virtual output queues. In oneimplementation VOQC 1428 receives an independent signal from backpressure logic 2312 corresponding to each virtual output queue (e.g.,one VOQ per tuple of physical output port, physical input port, andtraffic class). By forming the egress buffer and distributing circuit onone substrate, a large number of wired signal connections (e.g., for“go” signals from each buffer queue to back pressure logic) areeconomically and reliably implemented.

In one implementation, each segment is served by a plurality of channels(e.g., four to achieve a data rate up to four times the data rate of onechannel). The arbitration circuit for a virtual output queue may place aframe onto a selected one of the four channels. In an alternateimplementation each channel has an arbitration circuit that servesvirtual output queues (e.g., seventy two queues being four trafficclasses times eighteen source port identifiers). A particular virtualoutput queue may be served by more than one arbitration circuit.

Routing information may be stored local to one routing processor andmessages to be routed using that information may be routed from otherrouting processors via the fabric to that routing processor. Forexample, router 105 of FIG. 24 includes routing processors 2402 and 2404each as discussed above with reference to routing processor 1161. Eachprocessor has access to memory for a virtual context table not used bythe other processor. Routing processor 2402 includes memory for virtualcontext table 2403; and, routing processor 2404 includes memory forvirtual context table 2405. Virtual context tables (VCT) 2403 and 2405may be stored in memory on the same integrated circuit substrate as therespective routing processor (e.g., an integrated circuit implementationof a port logic circuit) or may be stored in a memory circuit havingareas reserved for access by each processor (e.g., portions of memorycircuit 1162 as discussed above with reference to Table 12). Routingprocessors 2402 and 2404 route packets via fabric 2406 (e.g., asdiscussed above with reference to fabric 213) using a fabric frame thatencloses the frame used on network 101. The enclosing fabric frameheader may include a designation indicating one of the following:(type 1) the receiving routing processor is to perform no framemodification; (type 2) the receiving routing processor is to performvirtual to nonvirtual frame modification; or (type 3) the receivingrouting processor is to perform nonvirtual to virtual framemodification. The frame modifications for types 2 and 3 above areperformed in the egress buffer of the receiving processor before theframe is transmitted onto network 101.

Use of fabric frame headers as discussed above is described by a seriesof messages 2400 of FIG. 24 that includes routing of virtual R/W I/Os tononvirtual R/W I/Os and vice versa as discussed above, for example, withreference to FIGS. 6-11, 13, 14, and 16-20. In message sequences 2400member 116 reads and writes a portion of a virtual resource implementedas nonvirtual resource 177 or member 115 (all of FIG. 1). A transactionthat includes an FCP_CMND sequence 2410, one or more pairs ofFCP_XFER_RDY and RD_DATA sequences 2420 and 2440, and an FCP_RSPsequence 2450 accomplish a read transfer of data from nonvirtualresource 177 to member 116. A transaction that includes an FCP_CMNDsequence 2410, one or more pairs of FCP_XFER_RDY and WR_DATA sequences2420 and 2430, and an FCP_RSP sequence 2450 accomplish a write transferof data from member 116 to nonvirtual resource 177. Messages “A” at time2411, “F” at time 2423, “G” at time 2431, “L” at time 2443, and “O” attime 2453 convey no identity of the nonvirtual entity on which the readand write operations occur. Messages “C” at time 2413, “D” at time 2421,“I” at time 2433, “J” at time 2441, and “M” at time 2451 appear to theresource as nonvirtual network traffic with no indication (other thanthe network address of the proxy) that the initiator is a proxy asopposed to a nonvirtual member. Field values used in routing messages ofseries 2400 are described in Tables 22 and 23.

When VCT 2403 has routing information for the transaction identified inmessage “A” at time 2411 as a virtual transaction from member 116 inFCP_CMND 2410, messages “B” at time 2412 and “H” at time 2432 are markedby routing processor 2402 as type 1. Routing processor 2404 in itsegress buffer (e.g., 1414) removes the marking and passes the payload asmessages “C” at time 2413 and “I” at time 2433.

When VCT 2405 does not have routing information for the virtualtransaction of message “A”, routing processor 2404 marks messages “E” attime 2422, “K” at time 2442, and “N” at time 2452 as type 3 (e.g.,1730). In response, routing processor 2402 performs modification to eachframe in its egress buffer (e.g., 1910).

In an alternate configuration wherein VCT 2405 has routing informationfor the transaction identified in message “A” at time 2411 as a virtualtransaction and VCT 2403 does not, routing processor 2402 marks messages“B” and “H” as type 2 and receives messages “E”, “K”, and “N” marked byrouting processor 2404 as type 1. The processing burden of performingframe modifications in ingress and egress buffers may be allocated by anadministrating process (e.g., managing virtualization). Allocation andreallocation may be accomplished as discussed above with reference toflags returned from a virtual flow lookup in Table 11.

TABLE 22 Field Values Mes- LUN and sage Frame Type S_ID D_ID OX_ID RX_IDLBA A FCP_CMND I VM IX — VR B VM T PX — NR C VM T PX — NR D FCP_XFER_(—)T VM PX TX — E RDY T VM PX TX — F VM I IX PX — G WR_DATA I VM IX PX — HVM T PX TX — I VM T PX TX — J RD_DATA T VM PX TX — K T VM PX TX — L VM IIX PX — M FCP_RSP T VM PX TX — N T VM PX TX — O VM I IX PX —

TABLE 23 Field Value Meaning Assigned By Description I Initiator networkport Manufacturer of the WWPN for initiator (e.g., 167). identifierInitiator system T Target network port Administration WWPN for target(e.g., 115). identifier VM Virtual member Administration when A networkport identifier identifier designing zones intercepted by a routeroperating according to various aspects of the present invention. Forexample, an address in a range of addresses that are reserved fordesignating the router. VR Virtual resource Administration when Aresource logical unit identifier identifier designing zones (e.g., LUN)that has no corresponding physical entity. NR Nonvirtual resourceManufacturer of the WWPN for actual LUN (e.g., 177). identifier Targetsystem IX Initiator's transaction Initiator Any transaction identifiernot identifier currently associated with this initiator. TX Target'sexchange Target Any transaction identifier not identifier currentlyassociated with this target. PX Proxy's exchange Routing processor Anytransaction identifier not identifier currently associated with thisproxy.

The foregoing description discusses preferred embodiments of the presentinvention which may be changed or modified without departing from thescope of the present invention as defined in the claims. While for thesake of clarity of description, several specific embodiments of theinvention have been described, the scope of the invention is intended tobe measured by the claims as set forth below.

1. A method performed by a router for routing frames in a network, themethod comprising: a step for receiving routing information at amanaging processor of the router, the routing information comprising aplurality of tuples, each tuple comprising indicia of a virtual entityand indicia of a nonvirtual entity; a step for preparing at the managingprocessor a routing table in accordance with the routing information; astep for providing the routing table to a supervising processor of therouter for storage in a memory circuit of the router, the supervisingprocessor coupled for data transfer between the managing processor andthe memory circuit, the memory circuit being accessible by a pluralityof routing processors of the router coupled by a fabric of the routerfor routing frames to another routing processor of the plurality via thefabric; a step for receiving at a routing processor of the plurality aframe of a virtual transaction; a step for accessing a tuple of therouting table at the routing processor; a step for redirecting the frameto a nonvirtual entity in accordance with the accessed tuple of therouting table; and a step for routing the redirected frame.
 2. Themethod of claim 1 wherein the indicia of a virtual entity identifies atleast one of a virtual participant, a virtual member, a virtualresource, a virtual device, a virtual address, a virtual page, and avirtual sector.
 3. The method of claim 1 wherein the indicia of anonvirtual entity identifies at least one of a nonvirtual participant, anonvirtual member, a nonvirtual resource, a nonvirtual device, anonvirtual address, a nonvirtual page, and a nonvirtual sector.
 4. Themethod of claim 1 wherein the nonvirtual entity comprises a proxyprocess performed by the managing processor.
 5. The method of claim 1wherein the nonvirtual entity comprises a nonvirtual resource of amember of the network.
 6. The method of claim 1 wherein the step forproviding further comprises a step for transferring the routing tablefrom the managing processor to the supervising processor via a localarea network coupling the managing processor to the supervisingprocessor.
 7. The method of claim 1 wherein the step for redirectingcomprises at least one of: a step for rewriting a source identifier; anda step for rewriting a destination identifier.
 8. The method of claim 1wherein routing information comprises a policy value in association witha virtual resource identifier; and the step for routing the redirectedframe comprises a step for routing in accordance with the policy value.9. A method for distributing the burden of virtualization among routersof a network, the method comprising: a step for receiving first routinginformation at first router; a step for receiving second routinginformation at a second router; a step for receiving at the first routera payload of a first frame; a step for determining that the first frameis part of a nonvirtual transaction; a step for routing the payload tothe second router as part of a nonvirtual transaction; a step forreceiving at the second router the payload; a step for determining thatthe payload is part of a virtual transaction; a step for accessing atuple of the second routing information; a step for redirecting thepayload to a nonvirtual entity in accordance with the accessed tuple ofthe routing information; and a step for routing the redirected payload.10. The method of claim 9 wherein the second routing informationcomprises a policy value in association with a virtual resourceidentifier; and the step for routing the redirected payload comprises astep for routing in accordance with the policy value.
 11. A router forrouting frames in a network, the router comprising: a memory circuit; aplurality of routing processors, coupled by the fabric for routingframes to another routing processor of the plurality via the fabric, thememory circuit being accessible by each routing processor of theplurality; a managing processor; a supervising processor coupled fordata transfer between the managing processor and the memory circuit;means for receiving routing information, the routing informationcomprising a plurality of tuples, each tuple comprising indicia of avirtual entity and indicia of a nonvirtual entity; means for preparing arouting table in accordance with the routing information; means forproviding the routing table to the supervising processor for storage inthe memory circuit; means for receiving at a routing processor of theplurality a frame of a virtual transaction; means for accessing a tupleof the routing table at the routing processor; means for redirecting theframe to a nonvirtual entity in accordance with the accessed tuple ofthe routing table; and means for routing the redirected frame.
 12. Therouter of claim 11 wherein the indicia of a virtual entity identifies atleast one of a virtual participant, a virtual member, a virtualresource, a virtual device, a virtual address, a virtual page, and avirtual sector.
 13. The router of claim 11 wherein the indicia of anonvirtual entity identifies at least one of a nonvirtual participant, anonvirtual member, a nonvirtual resource, a nonvirtual device, anonvirtual address, a nonvirtual page, and a nonvirtual sector.
 14. Therouter of claim 11 wherein the nonvirtual entity comprises a proxyprocess performed by the managing processor.
 15. The router of claim 11wherein the nonvirtual entity comprises a nonvirtual resource of amember of the network.
 16. The router of claim 11 wherein the means forproviding further comprises means for transferring the routing tablefrom the managing processor to the supervising processor via a localarea network coupling the managing processor to the supervisingprocessor.
 17. The router of claim 11 wherein the means for redirectingcomprises at least one of: means for rewriting a source identifier; andmeans for rewriting a destination identifier.
 18. The router of claim 11wherein routing information comprises a policy value in association witha virtual resource identifier; and the means for routing the redirectedframe comprises means for routing in accordance with the policy value.19. A network for distributed virtualization, the network comprising:means for receiving first routing information at a first router; meansfor receiving second routing information at a second router; means forreceiving at the first router a payload of a first frame; means fordetermining that the first frame is part of a nonvirtual transaction;means for routing the payload to the second router as part of anonvirtual transaction; means for receiving at the second router thepayload; means for determining that the payload is part of a virtualtransaction; means for accessing a tuple of the second routinginformation; means for redirecting the payload to a nonvirtual entity inaccordance with the accessed tuple of the routing information; and meansfor routing the redirected payload.
 20. The network of claim 19 whereinthe second routing information comprises a policy value in associationwith a virtual resource identifier; and the means for routing theredirected payload comprises means for routing in accordance with thepolicy value.